Automatic plant watering system

I am away from home for three months. I wanted a way to ensure that two of my plants could survive while I was away, without a human watering them for a long time.

I built a small system to automate the plant watering process. Here is a picture of the system in action:

Waterbot Still

Some nice features of my watering system are:

  • Watering is done on a fixed schedule, but I can log in remotely to adjust the schedule.
  • Soil moisture is logged and moisture graphs are periodically generated.
  • A camera takes a picture of the watering system twice an hour.
  • Surveillance pictures and moisture graphs are uploaded to a website so I can easily check the status of the system.

Here are examples of the moisture graphs that my system generates:

m2

m1

The moisture graphs are generated using the R programming language and the excellent ggplot2 library. Here is a snippet of the code that generates one of the graphs above:

tod <- function(x) as.numeric(format(x, "%H")) + as.numeric(format(x, "%M"))/60
ggplot(moist, aes(x=time, y=m1)) + geom_point(aes(color=tod(time))) +
    scale_color_gradient(low='blue', high='orange') +
    labs(title='Plant 1', y='moisture')

One of the challenges of designing the watering system was that I had not done any hobby electronics projects before, so I had to learn a bit of electronics basics and refresh my memory from my undergrad electronics class.

Arduino and Raspberry

I used an Arduino Uno to prototype the water pump control. I had a lot of fun using the Arduino, and I ended up doing several other projects with Arduinos as well, such as reverse engineering an IR remote control for my home audio system.

The final watering system uses an Arduino to control a submersible water pump, and to measure soil moisture using two moisture sensing devices. The Arduino uses a USB serial port to listen for commands from a Raspberry Pi which has scripts that determine the watering schedule. This setup keeps the code on the Arduino really simple, and allows me to tweak a lot of the logic in the system by logging in to the Raspberry Pi via a remote shell. I could even reprogram the Arduino remotely if I wanted to.

The Raspberry Pi has a camera for taking the surveillance pictures. I’m planning to use the images from the camera to compile a time lapse video later on.

Here is a closeup of the box that sits on top of the water reservoir. It contains the Arduino and a prototype board with the power switching circuit for the pump:

Control Box

Tuning

I have only one water pump that waters two separate plants. To make this work I have a regular garden hose divider to split the lab tubing and water both plants at the same time. I tweaked the length of the tubes that lead to the two plants to adjust the amount of water going to each plant. To make adjustments without drowning my plants I used plastic bags to collect the water while I was tweaking the system. Collecting the water in plastic bags let me measure exactly how much water each plant was receiving.

The current watering system is not flexible and relies on the two plants consuming relatively the same amount of water. If I decide to improve the watering system in the future I will probably install solenoid valves so that I can water each plant independently of the other. This would make the system more reliable for long-term use and make it possible to use a separate control loop for each plant.

Flooding concerns

Flooding was a big concern when I designed the system, so I placed each plant in a big plastic container large enough to collect all water in the reservoir in case a catastrophic failure should happen.

The towel under the water reservoir is there to catch condensation from the outside of the reservoir. I did not notice any condensation problems while the system was in trial mode before I left. Condensation could be a problem if the room temperature changes faster than the reservoir temperature. One other option I considered was to place the reservoir in another container like the one the big plant is standing in.

Future work

As mentioned, I would like to have individual plant watering using solenoid valves. This could also make the system modular – I could add extra plants to the water loop and have them individually watered just by adding an extra piece of tubing and a valve for each plant.

Another way of improving the system is to add moisture feedback in the control loop, and use e.g. a PID regulator to control the water pump. It is probably good to have oscillating moisture levels, at least that seems more natural for the plants, but I’d like a regulator that ensures that the three-day average moisture stays the same over a long time.

Coding Zen: Don’t Catch Exceptions

Usually the right thing to do with exceptions is to not catch them. This may seem counter-intuitive, so I will try to describe why I think it is a sound idea.

Consider this piece of Python code:

try:
    work()
except:
    print "Unhandled exception"
    sys.exit(1)

The code above prints an error message and halts execution if an exception occurs. This is a pretty common pattern, after all it is usually not fun to have stack traces printed on the console, which is the default behaviour when an exception goes uncaught. For example, here is the stack trace Python prints for the exception thrown by pressing Ctrl+C:

^CTraceback (most recent call last):
  File "test.py", line 6, in <module>
    work()
  File "test.py", line 4, in work
    time.sleep(30)
KeyboardInterrupt

The stack trace looks scary, like something broke horribly. For that reason we don’t want to expose stack traces to users, so it’s common to avoid the stack trace by using the previous code pattern to catch all exceptions.

I will now try to show why the above pattern, and the idea of eagerly catching exceptions, is harmful.

Obscuring error causes

When an exception is caught the responsibility to inform the user of what went wrong is on the code that catches the exception. If the catching code prints a generic error message we won’t know what caused the error. The worst thing you can do is to not print anything at all – that will leave the user not knowing if anything went wrong and that leads to a more frustrating experience than if they clearly know something went wrong.

Even if the exception catching code prints an error, you have to include enough information to be able to debug the error later. If your code does a lot of stuff which may cause exceptions, how do you know which part caused the program to stop and print “Unhandled exception”?

Catching exceptions with too large nets causes problems by obscuring the cause of errors in the code. The code below is harder than necessary to debug:

try:
    work()
except Exception as e:
    print e

Even though we print the exception it may be difficult to track down the cause. The print statement only prints the exception message, not the location it was thrown. For example, if there is a divide-by-zero error somewhere in work(), or the code called by work(), then the above code prints out integer division or modulo by zero. That’s not very nice, especially when you get this in a bug report from a user. How will you track down the location? It was probably a bug that triggers seldom, under conditions that don’t happen in your test environment. It would be much more useful to get the stack trace which clearly includes the location of the error:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    work()
  File "test.py", line 4, in work
    1/0
ZeroDivisionError: integer division or modulo by zero

If your program will be run by non-technical users it might be a good idea to write the stack trace to a file which they can attach in a bug report. Catching exceptions so that they are logged with the stack trace is a fine approach, but the key idea here is that exceptions should be allowed to propagate up to a handler that does more than just say “something went wrong”.

Don’t catch all exceptions

In most cases, it is not worth catching exceptions. Rather than having to convince yourself that you should not catch an exception, you should convince yourself that you should catch it. This mindset is important even in languages that enforce stricter handling of exceptions, like Java:

private void work() throws IOException { ... }
void doSomething() { work(); }

Should the doSomething() method above declare IOException to be thrown, or should it handle the exception? The compiler forces you to pick one. I would argue that you should let the exception be thrown, and not catch it, unless you have a good reason to catch the exception.

Which suggested fix would you choose?

In Java, unfortunately, programmers are conditioned to handle the exception locally because adding a new exception to a method signature often means that we just have to make the same decision in potentially multiple other locations in the code. Thus, sloppy exception handling is common in Java. I’ve often seen code that just adds a catch clause to get rid of a compile error.

I have had to solve many bugs in Java code, and some of those bugs were needlessly obscured by over-eager exception catching. The problem is not exclusive to Java, but I think it is more common in Java than Python.

When to handle exceptions

There do exist good reasons to catch exceptions, I just mean that we should convince ourselves that we really need to catch the exception before we blindly add code to catch it. Writing code without thinking it through is pointless busywork, and often leads to stupid mistakes. I notice myself falling in this trap constantly.

One good reason to catch an exception is if the operation that caused the exception should be retried. This is only feasible for certain combinations of operations and exceptions, so in those cases only specific exceptions should be caught.

Two examples of operations that can/should be retried are:

  • Downloading a file when an network error occurs.
  • Asking a user for a password and it was incorrect.

Another good reason to catch an exception is if the exception is benign or if the operation was an optional step that is safe to skip. Again, it is important to make sure that only the specific exception to be suppressed is caught.

It is not a good idea to catch more exceptions than needed, and the worst case is to indiscriminately catch all exceptions. Catch only the ones that you know you want to handle!

Let’s go back to the first example with the Python code and for instance say that we want to not print the stack trace when the user hits Ctrl+C – the program should just silently exit. That’s fine, but keep in mind that it is important to ensure that we don’t hide serious errors by catching too many exceptions. The following code solves the problem by catching only KeyboardInterrupt exceptions while letting other exceptions pass through:

try:
    work()
except KeyboardInterrupt:
    print "Unhandled exception"
    sys.exit(1)

Providing additional info

It is sometimes useful to provide some additional context for an exception by catching it and then re-throwing it. For example:

try:
    work()
except:
    print "Something went wrong while doing the work"
    raise

This is fine, the exception handler may even provide a useful bit of contextual information that can help later debugging. I don’t really think of this as catching an exception though, since it is re-thrown at the end of the exception handling code.

The Zen

The final Zen of the matter with exceptions is that the default should be to not handle exceptions, and only catch the ones that there is a specific reason to catch.

Not catching exceptions can simplify your code. It removes a few lines for exception handlers, simplifies the control flow, and gives peace of mind that some bugs are not needlessly obscured.

My Thoughts on JavaFX

In my previous post I mentioned the idea of using JavaFX for the Chunky GUI. Chunky currently uses the Swing toolkit for its GUI, and JavaFX is aiming to replace Swing eventually. I was interested in using JavaFX because it seems to have better cross-platform consistency. I have now done some exploratory development using JavaFX 8, and I want to share my thoughts so far.

What JavaFX does right

Consistency between platforms is the reason I wanted to try JavaFX. I made a clone of part of the Chunky GUI in JavaFX and tried running it on Windows 10, OSX, and Linux, and it looked the same on each OS. That’s awesome, it’s exactly what I wanted. Here is a screenshot of the GUI running on Windows 10:

Chunky JavaFX GUI

Another great thing about JavaFX is that it’s possible to separate layout from behavior. In Swing the only way to build a GUI was to write code that constructs each element manually. There did exist layout editors for Swing, but they all behaved differently and they generated ugly code that got mixed with the application code. Using Swing layout editors was sometimes a useful first step when building a GUI, but I always ended up having to code everything by hand because after I had made changes to the generated code the layout editor no longer could edit the layout. I really hate working with Swing layouts by manual coding but that’s what I had to do mostly.

JavaFX solves the layout editing problem by using XML files to define the layout. The layout is edited with a program called Scene Builder. Separating the layout from the code in this way means that you can continue using the layout program even after you edit the code – because they are separate. It is still possible to edit GUI elements via code if you need to, but it no longer has to be the main way to construct a GUI. Graphical editing is so much better for building GUIs! Here is a screenshot of the Chunky GUI in Scene Builder:

Chunky UI in Scene Builder

Although it’s nice to remove the manual coding of GUI layouts, it is of course necessary to wire up the GUI components to code that can make them do useful things. Coding with JavaFX is easier than with Swing thanks to a much nicer API. For example:

  • JavaFX uses functional interfaces for all callbacks which makes it easy to set up callbacks with lambdas in Java 8. In contrast Swing callbacks are grouped into listeners with multiple callbacks per listener.
  • Listening for modifier keys is supported in JavaFX with proper focus handling. In Swing you have to handle modifier keys for the whole application.
  • It is very easy to bind variables in the application to JavaFX GUI elements by wrapping the value in a JavaFX property.

A few other points about JavaFX that I like:

  • Customizable tooltips that can be built individually and placed anywhere.
  • Much improved color pickers.
  • Simpler file chooser API.

Things JavaFX needs to improve

My overall impression of JavaFX so far is quite positive, but I do have some gripes about JavaFX which I hope get fixed in the future. Below are three things that I hope will be changed/improved.

API additions between major Java releases

Some new API methods have been added to JavaFX 8 after the Java 8 release. For example you can call ChoiceBox.setOnAction() in Java 8 update 60, but it won’t compile or run on older Java 8 version. Adding things like this in Java updates means developers like me can end up unknowingly relying on a specific Java 8 release. I hate this and it’s a stupid idea. One of the cornerstones of Java is backward compatibility, and it should not be jeopardized for small API improvements. APIs must be stable so that you can rely on them. There is nothing more annoying than compiling a Java 8 program, and then trying to run it on another machine with Java 8 installed only to find out that it doesn’t work!

No image scaling without interpolation

JavaFX provides a drawImage method that can scale images, but it does so by using a blurring interpolation mode and there is no way to change the interpolation mode. This causes undesirable blurring when the image is enlarged. In Swing this worked just the way I wanted, but in JavaFX I have to implement my own scaling code just to fix this annoying behavior.

Scaled image in JavaFX - a blurry mess.

Scaled image in JavaFX – a blurry mess.

Lag

Feedback is very important in a user interface. The feedback needs to be delivered as fast as possible. In JavaFX when I hover over a button it takes a short while before the highlight that indicates the button is hovered appears. This may seem like a nitpicky complaint, but it is important to me that GUIs are responsive and right now JavaFX on Windows does not feel responsive enough. On the other hand I have noticed a huge performance regression in Swing based GUIs on Windows 10 recently making them lag ridiculously much. At least JavaFX is a little better than Swing on Windows 10 right in that respect.

Chunky 2016 Spring Update

In January I published a small snapshot release that added more posing options for player entities. My plan was to improve the posing support and also improve player rendering before I made the 1.3.8 release, however I became very busy with my PhD studies/work and Chunky development has been mostly on hiatus since then. I wanted to write this post to give an update to the Chunky development to everyone who uses Chunky and is interested in the progress of development.

Release 1.3.8

Since the improvements I wanted to add before the 1.3.8 release got delayed, I think it will be a good idea to make the 1.3.8 release anyway because I have made some bugfixes that need to be released. Today I published a new snapshot release, 1.3.8-rc1, which is a release candidate for 1.3.8. If I don’t discover any new issues with this snapshot release I will publish that version as 1.3.8.

User Interface plans

Of all the components in Chunky, the User Interface (UI) is the one I have spent most time on. I actually very much dislike UI programming. It’s tedious, error prone, and time consuming. I am still not pleased with the UI, and there are many things I’d like to improve in it. The most annoying issue right now is that the UI looks and behaves differently on different platforms.

Chunky uses the Java Swing library for its UI components. There are many benefits of using Swing: it is supported on most Java versions and it works on Windows, Mac, and Linux. However, the disadvantages of Swing are also many, and chief among the disadvantages is that the UI components are ugly and behave in some cases unexpectedly. Swing UIs usually end up looking old and janky. The default Swing look and feel is very outdated, but people have tried to remedy this by making their own look and feel libraries for Swing. This is a huge time sink and not at all worth it for small projects like Chunky.

Swing does provide platform-dependent look and feel settings which allow your Java app to look more like a native app, and these settings can make Swing look much better, however it also means that the UI looks different on each platform, and behaves slightly differently. For example, here is a side by side comparison of the control panel from the main window as seen on Windows and Linux:

Swing on Windows vs Linux

I am very displeased with the current state of the Chunky UI, and I’d like to remove these large differences between platforms. I would much prefer if the UI looked the same on all supported platforms. There is a solution to this problem, and that is the Java FX framework. Using Java FX would require users to use Java 8, which I dislike because I really would prefer to support Java 6 as long as possible, however Java FX applications can look really pretty, and they behave well and are consistent between platforms! My plan right now is to try to migrate to Java FX after the 1.3.8 release. Swing components can be used in Java FX and that will allow me to gradually move over to Java FX.

The downside of making a big change in the UI is that it takes a lot of time, and it’s development time that doesn’t help improve the application in a substantial way. In some cases a better working UI can be a big gain, but it won’t add new features or fix existing rendering bugs. So it is difficult to commit to a big UI rework when I know that it detracts from other fixes/improvements that I would like to work on.

Single Document Interface

I had an idea previously to design a single document interface for Chunky. I started working on that, using Swing, but decided to ice the project when I realized that it would take a lot of time to iron out all of the small new issues that were introduced. I still think I’d like to try making a single-document interface in Java FX. Here is a screen shot of one of the partially working single document interface builds:

Single document interface in Chunky

Development hiatus

As mentioned, Chunky development has been mostly frozen for a while. This is due to my PhD studies and all the work I need to do because of it. There will be another hiatus in Chunky development soon, during the summer, because I am returning to Google for a second internship. After that, in August, I am looking forward to taking some vacation time and focusing on Chunky development.

Printf debugging is useful

Debugging is an essential skill for programmers. Our code breaks often, and in many cases we have to debug our code to figure out why it broke. Some people hate debugging while others love it. I am firmly in the “love debugging” camp, I get a huge kick out of solving a bug in my code by figuring out what’s going wrong and why.

When I think about debugging, printf debugging is usually the first thing that comes to mind, though many may not see it as real debugging. In this post I will try to describe what printf debugging is and why I think it is useful.

First, let’s review some common debugging techniques:

  • Looking at the output. Some kinds of errors can be found without even looking at the source code. This is common whenever you are working with graphical applications, e.g. in a Graphical User Interface it is often easy to identify a bug and how to fix it when something doesn’t work as intended. Just looking at the program output in non-graphical applications can be useful too. For example, if your program should print a list of items and it prints them in an unexpected order, then the bug is probably that the list is not sorted before being printed.
  • Stare at it until you figure out what when went wrong. This is a surprisingly common debugging technique. You can often figure out what is wrong by just reviewing your code again and thinking about how the control flow could lead to the unintended behavior.
  • Explain your code to someone else (or a rubber duck). This deubgging method is useful but often overlooked. It is easy to make incorrect assumptions about your code, it’s a mistake every programmer does many many times. When you try to explain your code to someone else something interesting happens: you end up re-evaluating your assumptions and realizing that some of them were wrong, and that can lead to discovering the cause of a bug.
  • Printf debugging: inserting print statements in your code to test your assumptions or visualize control flow. We’ll talk more about this soon.
  • Using a debugger. This is probably what people associate most with “real” debugging, although it may not be the most common method because it requires additional tools and mental context switching. Debuggers can be hugely powerful though, and there are lots of different ways to use a debugger ranging from simple to advanced: stepping through instructions, setting breakpoints, setting watchpoints, setting conditional breakpoints / watchpoints, attaching to a running program. Knowing how and when to use a debugger is awesome, but debuggers are not the main topic of this post.

Let’s examine printf debugging a bit more. As I said, printf debugging is the practice of inserting print statements in your code. You might not even think of this as a debugging technique because it doesn’t rely on a debugger, but as seen in the list above many debugging techniques don’t require a debugger. The lack of a debugger is in fact a part of why printf debugging is useful. Although many development environments come with a built-in debugger, they usually require a separate workflow to start the debugger. It might not be that difficult, it’s usually quite easy, but it is often just quicker or simpler to insert a few print statements to test your code. I usually find that inserting print statements is less mentally straining than switching to a debugging mode – it means I have to switch the way I think about my code and figure out useful places to add breakpoints.

Here are a few other reasons why I think printf debugging is useful:

  • You don’t need extra tools to insert print statements. Just edit the code then build and run the program as usual.
  • Print statements can be used to follow the control flow of the code, without stepping through it in a debugger. Using print statements you can easily generate a timeline of of interesting points in the execution. Selected variable values can also be printed in the same high-level timeline if needed.
  • Low impact on timing sensitive bugs. Using a debugger interrupts the control flow and can cause timing-sensitive issues to become untraceable. Print statements are less intrusive when debugging concurrent code that depends on a specific thread interleaving to be triggered.
  • The printf method usually gives more high-level information than using a lower-level technique tool such as a debugger where you often step through statement-to-statement and see very detailed but often uninteresting information about the control flow and program state.
  • A debugger may not be available, then printf debugging might be the best bet to figure out what is actually happening in the code.

Printf debugging does have lots of legitimate uses, however in some cases it’s not the best thing to do. Here are some reasons why printf debugging can be bad in some situations:

  • The debug print statements need to be removed after you fixed the issue. This is tedious and error prone. It’s easy to forget to remove a debug print statement.
  • It’s boring to insert print statements. When there are many variables you want to know the value of it is probably time to switch to using a debugger.
  • Output logging is not always available. If you make apps for a smartphone it might not be easy to get a console to print debug messages on. Many kinds of plugins in different kinds of software don’t have a debug console either.
  • Too much output. Printf debugging is sometimes too verbose, particularly when the place you insert the print statements is called very many times before the bug occurs.

I think there is much more that could be discussed about the pros and cons of printf debugging, but the main point is that it’s a simple technique to use and there is no shame in using a less sophisticated tool if it gets the job done. In the end it only matters if you can find the bug in a reasonable time, not which tools you used to find it.

One final thing I want to show is a technique I use when I debug Java code. Sometimes I add the following line to my code in a place where I know that incorrect behaviour is happening:

new Throwable().printStackTrace();

This will result in a stack trace showing the call stack for the problematic point in the program. This starts to get into territory where it’s better to use a debugger, but if you just want to know the location of one statement in the call stack and are not bothered to inspect local variables etc, then the above line might be a useful hack.

Loving The Witness

I have been playing The Witness recently. It’s a gorgeous puzzle game. I am a fan of puzzle games and I think The Witness is one of the best I have played so far. The puzzles are what really make The Witness great, but the aesthetics are a big plus too. The atmosphere of the game is somewhat like Riven, in that you explore a desolate island world listening to gentle gusts of wind, rustling leaves and lapping waves, getting mildly infuriated by puzzles you don’t quite get. While not being vexed by an inscrutable puzzle, it can be quite meditative to walk around in the serene game world of The Witness.

The Witness

It is difficult to talk about the puzzles in The Witness without spoiling the game, and I really encourage you to try it if you are at all interested in puzzle games. I will try to describe why I love the game without spoilers.

A puzzle game needs a progression of difficulty – the puzzles can’t be straight up super hard, that would make it feel unfair and make players give up prematurely. It also can’t be too easy all the time, because making the puzzles too easy takes away the rewarding feeling of solving the puzzles, and then the game becomes boring – a tedious task rather than a rewarding challenge. I have played many games that were too easy and became boring because of that, for example The Bridge is a decent game but a bit too easy for me so I didn’t enjoy it as much.

In my opinion, the puzzles in The Witness are well balanced. They constantly offer new challenges by repeating old mechanics in slightly different ways. The Witness may be too difficult to some players. It does get quite complicated the further into the game you get, but it is on the other hand really rewarding to finally solve those difficult puzzles. Sometimes you just need to take a break from the game, or go look at another set of puzzles while you think about the puzzle you were stuck on in the back of your mind.

The Witness

Another great aspect of puzzle games is when you can solve the puzzle without being told how to. The best puzzle games manage to hint at the solution without blatantly telling you the solution. Solving puzzles like that feels really great – it makes you feel smart, and that is a big part of playing puzzle games. The Witness manages to give useful hints without being too obvious, and there are very few situations where the solution is directly given to you. I also feel like the hints are not too obtuse either, but there were definitely some puzzle mechanics that I had to just take a break and come back to to figure out the hint.

The Witness

The puzzles in The Witness involve many different mechanics, and it is up to the player to discover those mechanics. It’s all about figuring out what rules the puzzles obey, and which order to do certain things. The solution to the puzzle is really just moving a cursor from point A to B without violating those rules. It’s a simple system with a big variety of mechanics that challenges you with new and interesting head gymnastics. The game is also open-ended in this sense, because there are many puzzles at any time that you can try to solve, but some of the puzzles teach you how to solve others of the available puzzles.

A puzzle in The Witness

Though I really liked The Witness, there are some things that could be improved:

  • The game seems to be unplayable for colorblind people due to using colors heavily in the puzzles.
  • Some puzzles are more difficult or impossible for hearing impaired people.
  • The story is abstract and unsatisfying. I’d like to know more about what happened to the island and why I am there. It seems sort of like a virtual reality spa, but I guess it is up to interpretation.
  • The save system could be done a bit better. If you want to use multiple saves you have to trust the game to not overwrite all of your old auto saves.

The Witness may be a bit pricey right now, but it was worth the money for me. The highlights of the game are:

  • Rewarding puzzles that make you feel smart.
  • Pretty graphics.
  • Awesome atmosphere and sounds.
  • Lots of puzzles (more than 500!).
  • Lots of secrets.