Cover photo - contextualelectronics.com

tl;dr What DOES work

The 3 F’s

When I think of tooling, I think of my dad’s electronics shop, a big sprawling room, filled with random parts, power supplies and oscilloscopes. I used to spend my Saturdays with him as a kid. He would “put me to work” and I would help with small tasks, unsolder some old burnt-out capacitors, or log data on faulty power supplies. Nothing critical, but it was neat getting to learn and play at the same time. Anytime there was a big project around the house, my dad would rope me in. Fix a car, build a shed, my dad always had the right tool for the right job, form, fit and function.

… he would say. This was in stark contrast to an uncle I had, who would always go out and buy the absolute “best” and most expensive version of what my dad owned. Why buy a 14-inch chainsaw, when you can buy a 20-inch? I later learned that less is more, but god knows sometimes overkill is just fun. But here’s the thing, should work be “fun”? Does having that shiny new toy, make the process more enjoyable, or are you just showing off to the next person? It’s ok to want nice things, but how much, is too much?

Everything but

The friends and family with “the knack” tend to like their cars and trucks set up the way they like them, and desks the way they like them. Keyboards of all shapes and sizes, to suit each person. Perpetual tinkers, the lot of us.

As developers, we sit on the shoulders of GIANTS. As a result of this, it’s easy to get bogged down in the latest JS framework, or Python library. When all you have to do is pip install antigravity or npm, it’s easy to throw in everything and the kitchen sink at projects. Exploration is healthy, pushing boundaries, and learning new things. I (at times) have been very fond of chasing after the latest and greatest libraries when working on hobby projects. But there comes a point where how much, is too much. Do you NEED that extension, is that plugin going to HELP you? It’s easier to not think TOO hard about it and sink into the mindset of “Just another library, it’s less code I have to write…” or “This plugin seems helpful* for X”.

Projects like CookieCutter are great for setting up template-based repositories, for consultants in particular. A nice repeatable structure, that follows a known pattern, can save time and energy. There is a catch, as with all good things. A lot of smaller projects I’ve seen on GitHub dump heaps of neat toys, on otherwise simple codebases. Do I need (…checks notes…) twen… TWENTY-FOUR (I’m looking at you hypermodern-python) .. test modules for writing a library to turn a light switch on and off? The answer is no, not for anything this simple. For larger, more complex projects, maybe. For example, I use quite a few dev helpers on my personal projects, not because they are essential, but because they automate annoying tasks and let me think about my app, and not about structure (Black & Flake8) or security (Bandit, Tartufo & Whispers). (Shh, it’s 14 🙃)

I’ve lost count of how many plugins I have installed in VSCode, I’ll admit. Each one was curated over many years of finding things that suit my taste. The same goes with my custom-tailored PyCharm, a bit less so, due to the sheer genius of JetBrains’s developers thinking of so many things that Microsoft left up to the community. Open source vs closed source, give the people what they WANT, vs. give the people what you think they NEED.

There is no kill, like overkill, but sometimes, we lose sight of the problem we are trying to solve…

Too many options

People tend to be creatures of habit, finding comfort in the routine. This goes double (if not triple) for engineers of all trades. This idea is in stark contrast to the kitchen sink philosophy. I know what I know, and it gets the job done. Phrases like “I just want it done now” erode creativity. Tight deadlines, and angry clients for sure, make haste. Phrases like “it needed to get done yesterday”, can lower quality. There’s a phrase that comes to your mind.

“Jim Jarmusch once told me “Fast, Cheap, and Good…pick two. If it’s fast and cheap it won’t be good. If it’s cheap and good it won’t be fast. If it’s fast and good it won’t be cheap.” Fast, cheap, and good…pick (2) words to live by.”

-Tom Waits Spills the Beans to Tom Waits,” Sound Effects Blog, Boston Globe, May 22, 2008.

When we don’t give ourselves room to dabble, we can turn around three times and still be using COBOL… because it works (sorry if I’m being reductive). There is value sometimes to building a better mouse trap, but I’ve also seen folks fall prey to the “not invented here” syndrome. This battle of buy vs. build impacts us all. OpEx vs. CapEx, now vs. later, what’s the “best” answer?

We like what we like, and we like to do things the way we’ve always done them before because it works.

“So what do you do?”

I’ve been asked at many a family gathering, “What do you do?”. I usually quip back, “I’m a software developer” or “I work with the cloud”, two easy-to-digest answers. The honest answer, convert coffee into code… which gets converted into money by folks savvy in the ways of business. We’re in the business of solving problems, usually business problems, sometimes people problems, but always human problems.

(Side note: I did go to school for business… but I also was going to be a social worker as well. What you’re good at, and what you went to school for, are sometimes, very different things -:wink:)

Common ground & finding balance

“The most dangerous phrase in the language is ‘we’ve always done it this way’.”

Admiral Grace Hopper

While I’m primarily a Python (and PowerShell) developer, I know many folks out there use a variety of different languages, each with its specific tooling. What works for Python, may not work for Java C# or Go.

I’ve compiled a short list from talking amongst friends over the last few years. The whole point of this list is to be short, NOT exhaustive, and list flexible practices. That would defeat the core theme of this blog post if they weren’t! All of these tools work on OSX and Linux, and SHOULD also work on WSL if you are working on a Windows dev box. Please keep in mind, that all of these items are current as of 2023, and should be around for a fairly long time, nothing bleeding edge. If you don’t like the tools of today, considering making your own, just please do your research online first. We have had enough of a problem with competing standards.

Tools and patterns that work (IMHO) (Platform Agnostic)

• Git

  • This one goes without saying. While there are alternatives for hosting such as Bitbucket for native cloud-based hosting, products such as GitTea and GitLab, will always be close to my heart for locally hosted Git servers. One very sobering fact I’ve still seen, some CS programs STILL don’t teach this in CS-101. I would say Git is not hard, but that’s a lie 🙄
  • A great video overview on Git from MIT’s “Missing Semester” YouTube channel explains Git WAY better than I can in any blog post.
  • A quick note, with GitHub, you can still always run your own GitHub Actions as self-hosted runners if you run into issues with tests running fully cloud native.

• Docker

  • If you aren’t developing inside containers in 2023, you are just asking for “it works on my computer” problems. Solutions such as Vagrant aim to solve a similar problem, however, `docker run`` is about as simple as you can get while still giving you decent isolation. I would try and avoid running Docker on Windows, due to the requirement of needing Hyper-V. If you can run under WSL, you should have an easier time, but nothing beats NIX-based OS in this author’s opinion.

• Semantic Versioning

  • This versioning style allows for easily keeping track of how version updates will break builds during upgrades. Some vendors like to do date-style version numbers while some do not. Ex JetBrains IDEs vs VSCode.

  • Versions should be formatted like the below

      v1.2.3-beta+684
      Major.Minor.Patch-pre-release-type+build-number
    

    

    • Major: Incremented when incompatible API changes are made. Applications and other software that use the affected APIs will break. Hence, their code has to be updated.
    • Minor: Incremented when new functionality is added in a backward-compatible manner. It’s safe to update to a new minor version without requiring code changes. Code changes are needed only to make use of the new features.
    • Patch: Incremented when backward-compatible bug fixes are made. No new features are added. Some call this micro.

• Mermaid

  • A super handy way of putting diagrams in code. Many different IDEs and platforms support the format out of the box, and it is quite easy to read and write. PlantUML offers a more feature-rich alternative, a bit with less wide adoptions and harder (but more expressive) syntax.

  • Mermaid Example

    

    stateDiagram-v2
        [*] --> New_Idea
        New_Idea --> Working_On_Project:Write down idea
        Working_On_Project --> Bored:Life
        Bored --> New_Idea:Inspiration
        Project_Done --> [*] :Feeling of accomplishment
  

• EditorConfig

  • While this may not solve the age-old battle of tabs vs. spaces, you can at least enforce this standard in your repo and keep things consistent across IDEs and editors of all kinds. Just create a .editorconfig file at the base of your repo and you’re done!

  root = true
  [*]
  charset = utf-8
  end_of_line = lf
  indent_size = 4
  indent_style = space
  insert_final_newline = true
  trim_trailing_whitespace = true
  
  [*.md]
  indent_size = 2
  

• Tartufo

  • Great for scanning repositories locally for leaked keys and passwords. Works with several different languages.

  • Easy to use and deploy LOCALLY!

    pip install tartufo
    tartufo scan-local-repo . --progress
    

  • Can be integrated into your local Git pre-commit hooks, see below.

Tools and patterns that MIGHT work (Platform Agnostic)

• GitHub Actions
  • While there are as many ways to do GH actions as there are software on the internet, one set I keep on coming back to is to generate releases and auto add to-do’s. These YAML files go in your .github/workflows/ folder, inside your Git repo.

name: "Generate TODO to Issue"
on: ["push"]
jobs:
    build:
        runs-on: "ubuntu-latest"
        steps:
            - uses: "actions/checkout@v3"
            - name: "TODO to Issue"
              uses: "alstr/todo-to-issue-action@v4"
              with:
                  AUTO_ASSIGN: true

name: Release

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

on:
  push:
    tags:
      - "v[0-9]+.[0-9]+.[0-9]+"

jobs:
    release-gen:
        runs-on: ubuntu-latest
        permissions:
          contents: write
        steps:
        - uses: actions/checkout@v3
        - uses: ncipollo/release-action@v1
          with:
            allowUpdates: true
            generateReleaseNotes: true

    build-and-push-image:
      runs-on: ubuntu-latest
      permissions:
        contents: read
        packages: write
      steps:
        - name: Checkout
          uses: actions/checkout@v3
        - name: Docker meta
          id: meta
          uses: docker/metadata-action@v4
          with:
            images: ghcr.io/${{ env.IMAGE_NAME }}
        - name: Set up QEMU
          uses: docker/setup-qemu-action@v2
        - name: Set up Docker Buildx
          uses: docker/setup-buildx-action@v2
        - name: Log in to the Container registry
          uses: docker/login-action@v2
          with:
            registry: ghcr.io
            username: ${{ github.actor }}
            password: ${{ github.token }}
        - name: Build and push Docker image
          uses: docker/build-push-action@v4
          with:
            context: .
            platforms: linux/amd64 #,linux/arm64
            push: true
            tags: ${{ steps.meta.outputs.tags }}
            labels: ${{ steps.meta.outputs.labels }}
            cache-from: type=gha
            cache-to: type=gha,mode=max

• Conventional Commits

  • This is more of a way of writing commit messages, than a tool per se.

  • Messages are written in the format of “fix: foo” or “feat: bar”.

    git commit -m "fix: Thing no longer broken
    

  • How does this relate to SemVer?

    • fix type commits should be translated to PATCH releases (v2.3.4 -> 2.3.5).
    • feat-type commits should be translated to MINOR releases (v2.3.4 -> 2.4.0).
    • Commits with BREAKING CHANGE (ex. library function renames / backend data changes) in the commits, regardless of type, NEED to be translated to MAJOR releases (v2.3.4 -> 3.0.0).
    • If you play fast and loose with fix and feat and version numbers, PLEASE make sure you at least version-bump your Git tags for BREAKING changes, your teammates will thank you.

   | Type     | Description                                                    |
   | -------- | -------------------------------------------------------------- |
   | feat!    | Breaking change                                                |
   | feat     | New feature                                                    |
   | fix      | A bug fix                                                      |
   | revert   | Rollback of a change                                           |
   | docs     | Adding some documentation                                      |
   | ci       | Changes made to an infrastructure job                          |
   | chore    | Upgrades, cleaning up...                                       |
   | perf     | A performance improvement                                      |
   | refactor | A change in the code that does not affect the overall behavior |
   | style    | Formatting change                                              |
   | test     | Adding some test cases                                         |
  

• Change Log Generation

  This is a great extra included in the fantastic Git-Extras package (apt-get install git-extras OR brew install git-extras). Generating a nice change log is as easy as running git-changelog -p -a -x >> CHANGELOG.md at the base of your Git directory. As long as you have been tagging your releases with sane tag names, you should be good to go! While GitHub releases are always beneficial, this keeps your change log close to the code, just like your README file. Or do both with a little GitHub Actions love! Automate the boring stuff.

  n.n.n / 2023-04-22
  ==================
  
  * Added Auto change log generation
  * Removed radon from suite, not being used in interactive test
  * Added pause in test
  
  v2.0.0 / 2023-04-08
  ===================
  
  * Version bump
  * Merge branch 'feat/cli-interface'
  * Readme update
  * Added keyring check
  

• Commit hooks

  • Super handy for running tests and scripts on different Git actions. I tend to tie them to Task files and Make files, as it’s easier to update the file than the script. You can even keep it as simple as scanning your code base for secrets to prevent leaks!

• Postman

  • Great for testing and debugging API endpoints, a must-have for backend developers

  • For a simple CLI alternative, you might want to change out HTTPie (similar to curl)

• Make

  • While make files are more common among compiled languages, such as C++, I’ve been seeing a resurgence of them in the DevOps space, as an alternative to messy bash scripting and a simpler alternative to PyInvoke.

• Taskfile

  • A personal favorite of mine. Yes, I know we have Makefiles, but IMHO the syntax of this is easier to read and write. Before you get your pitchforks, please take a look, as I’ve said before, I try and keep things in this category both language-agnostic and cross-platform.

  • In this example, we can run task greet and we will load and print the vars from our env file and get the directory contents of our home directory.

    version: '3'
    
    env:
      ENV: testing
    
    tasks:
      greet:
        dir: /home/user
        dotenv: ['.env', '{{.ENV}}/.env.', '{{.HOME}}/.env']
        cmds:
          - echo "Using $KEYNAME and endpoint $ENDPOINT"
          - ls
    

• PyInvoke

  • This is a fantastic tool for when you need your make files to think! This build is done via a Taskfile…

    version: '3'
    vars:
        BUILD_DIR: pl_worker
    tasks:
        test:
            silent: false
            interactive: false
            dotenv: [prod.env]
            cmds:
                - poetry run vulture --min-confidence 100 {{.BUILD_DIR}}
                - poetry run xenon --max-absolute B --max-modules B --max-average B {{.BUILD_DIR}}
                - poetry run pytest --cov --cov-fail-under=75
    

    Can ALSO be done via PyInvoke

    from dotenv import load_dotenv
    load_dotenv("prod.env")
    from invoke import task
    
    BUILD_DIR= "pl_worker"
    
    @task
    def test(c):
        """Test Build"""
        c.run(f"poetry run vulture --min-confidence 100 {BUILD_DIR}",echo=True)
        c.run(f"poetry run xenon --max-absolute B --max-modules B --max-average B {BUILD_DIR}",echo=True)
        c.run("poetry run pytest --cov --cov-fail-under=75",echo=True)
    

    Will I redo all of my TaskFiles for my projects? No, what I have works for me and works well. But next time I need to write a CLI script runner for myself! (Side note: I’ve already written some neat stuff using Invoke for work… I like it 😄)

Plugins & Apps (Platform Specific)

While I could list off dozens of neat little things I’ve found over the last year or so, I’ll keep this list very short, just a few select plugins and apps that have really helped me be more productive. Please keep in mind that these are either IDE or platform-specific, so YMMV.

• AwesomeConsole

  Brings ctrl+click to the terminal Window, a-la VS Code

• ShiftShift

  On the flip side, brings the joy of a quick search from JetBrains to VS Code

• RayCast, Wox & ULauncer

  Depending on which platform you use, a quick desktop-wide search will save you hours over the months**. Really, check these out! (Yes OSX has the spotlight, and yes “it just works”, I would argue this is superior in every way to it)

Conclusions

While I know this blog post has been very VERY long, I do hope it helps those who come across it to not only improve their tooling but to also improve HOW they think about the tools they choose and choose NOT to use. I hope this blog post inspires those who read it to try some new toys… err tools!