Making Tools 2017-09-18

In The Mythical Man-Month, Fred Brooks describes his vision of an ideal software engineering team, called The Surgical Team. In this team, there is a dedicated toolmaker to help the lead programmer/system architect. The Surgical Team model hardly exists in practice nowadays. Instead, a software engineer is expected to fill multiple roles, including toolmaking. In my opinion, toolmaking is an essential skill for any productive programmer.

In my experience, it feels very fulfilling to building a tool to improve my workflow or solve a specific problem or need. I love programming because I love to solve problems, and building small tools is like an easier kind of programming with faster feedback. You also avoid a lot of the bureaucracy that comes with building a full application: writing tests, issue tracking, writing documentation, building releases, and so on. These things can be added later if the tool will be shared with others. However, I often make tools just for myself, and that simplifies the development process and makes it much more fast and fun.

When I think about programmer tools, I group them into three categories: simple one-off scripts, complex command-line applications, and GUI tools.

One-off Scripts

The most common type of tool that programmers build are small single-purpose scripts to perform a simple but repetitive task. Typically these small scripts are used to perform file-oriented tasks like batch renaming of files, uploading files to a server, or parsing data from a file and converting it into some other format.

Terminal emulators have simple scripting functions built-in so that you can compose scripts in a single command line. These are known as one-liners, and they often appear in Stack Overflow answers and tweets. There is even a website dedicated to Bash one-liners.

Simple scripts are great for automation. By using cron, you can make certain things happen at regular intervals. For example, I use cron scripts to regularly update websites when I commit to certain repositories. I also use cron to water my plants.

Command-line Tools

The natural evolution from a one-off script is to make a more complicated command-line utility. One of the main distinguishing factors is that command-line tools usually take input arguments, whereas a script usually does not.

A command-line tool may be a more reusable version of a script, that fits a more general purpose. In other cases, a command-line tool is used to solve more complex tasks that require much more code.

Interpreted languages like Python are popular for making command-line tools because they enable fast prototyping. With Python, you can test your tool without first compiling it. Python also has a REPL that allows testing small pieces of code quickly.

Here are some examples of tasks I have solved by making command-line tools:

  • Converting Markdown to HTML with additional template expansions.
  • Automated benchmarking for various applications.
  • Software release automation.

GUI Tools

Command-line tools can be very useful, but they are not ideal for data entry. Data can be provided to a tool via data files in a readable format, like JSON. However, for a tool that requires lots of user input it is often better to build a GUI for the tool. This enables modifying the input without restarting the tool.

It is not easy to build a GUI, especially if you are not familiar with a GUI toolkit. It may pay off in the long run to learn a GUI toolkit just to be able to build interactive tools for yourself. Simple web applications can work as a great platform to make tools with GUIs.

I often use JavaFx to build tools that require editable input. It is not the easiest toolkit to learn, but I use JavaFx in Chunky so I didn’t learn it specifically to make tools for myself. It works well for me because I prefer to use Java for most of my projects. Building tools in the same language makes it easy to integrate with the rest of the project.

I’ll show some examples of tools I made using JavaFx. The first one is a profiling tool:

This tool collects execution traces from a network socket. The data is displayed in interactive graphs, so that I can zoom in on interesting regions. The tracing data comes from applications that I want to profile. The tool helps me identify which parts of the applications take most time. The graphs give a great overview of the runtime behavior of an application.

The most recent GUI tool I made was a 3D model editor. I built this for Chunky development. In Chunky, all block models are made up of quadrilateral surfaces that are coded manually in Java. The spacial and texture coordinates of each surface must be precisely specified for it to render correctly. This works fine for simple models, but it becomes cumbersome for more complex models. For example, this is part of the code for the lever model:

The first, probably obvious, problem of using this format is figuring out which coordinates to plug in to the code. The second, and less obvious, problem is that I often have to restart Chunky to see what effect a change has. This takes a long time and is especially annoying when making tiny changes. It’s easy to get something wrong and have to tweak it multiple times, causing multiple restarts, to fix a problem.

To improve rapid prototyping I made a render testing tool that makes it easier to construct these 3D models in Java. The tool was written so that it could efficiently use hot code reloading in IntelliJ to reload code changes in a running instance of the tool. This helps a lot in building new models, but it was still very tedious to define 3D models manually in Java code. I usually resorted to drawing models on paper first to figure out the coordinates I needed.

Recently I was working on adding armor stand rendering, and I decided to build an interactive model editor to help with the modeling process. Here is a screenshot from the current version of the model editor:

The model editor is a great help in building new models, because I can immediately see the spacial relation between blocks and it takes away much of the tedium of manually defining quadrilaterals. The geometry in the editor is block-based, so the six individual faces of each block are automatically generated based on the parameters of one single block. Texturing the faces is also significantly easier using the tool, since I can directly see if any texture coordinate is incorrect.

Categories: Uncategorized

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.