Nov 25, 2021 8 min read paper club

What is Hammock Driven Development?

If the solution isn't obvious you don't understand the problem well enough.

This is a summary of Rich Hickey's 'Hammock Driven Development' talk. In it, he describes his process for thinking about and solving hard problems, a useful skill for programmers and non-programmers alike.

The talk is brilliant and I'd recommend that you stop reading this and listen to it first. I'm yet to meet someone that didn't take a loads away from it. If you're unable to listen, I hope you find some valuable pieces in this summary.

Why this talk is valuable

Learning to solve hard problems equips us to build better software. The word learning is used intentionally because it's a skill that can be practiced and acquired. Understanding the processes involved in problem-solving allows us to refine and improve our approach to it. This talk is about how to do that.

Core Ideas

The ability to solve difficult problems is directly correlated with the amount of time spent thinking about those problems.
Analysis and design is about understanding and solving the problems we're facing, not the actual implementation of solutions.
Solve problems, don't build features.
There is technique and skill involved with solving problems. It is something that can be learnt and improved. By practicing problem solving you will get good at it.
Problem-solving is one of the most leveraged skills you can acquire (more so than any tool, technology or framework)

The Rich Hickey Approach To Solving Hard Problems

In the talk, Hickey outlines the way that he thinks about problem solving and some of the methods that he's found useful. The ideas are malleable and should be adjusted to your circumstances and the problem you're trying to solve. Some of these steps will work for you, and some won't, but by deliberately trying to develop your problem-solving muscle, you're likely to see a lot of improvement.

1. State the problem

Say it out loud, talk with team member about it, write it down. Put it somewhere and talk about it. The process of talking about is important because it's the seed of solving the problem. It's easy to rush over this point, but don't. Take the time to articulate the problem you're solving.

2. Understand the problem

Take the time to understand the problem deeply. Go slow now so that you can go fast later. Some useful questions to begin with:

What do you know already (facts, context, constraints)?
What don’t you know? For example, if you know you will need input data, where does it come from? If the primary source goes down, is there a secondary?
Are there related problems? Hint: there are. Your problem is almost certainly not unique in the history of the world. Understanding the overlap between your problem and related problems is amongst the fastest way to find a solution.

3. Be discerning, be critical. Not everything is awesome, and it doesn't have to be

Look for defects in your proposed solutions and try to solve those right up front.

Do you think there aren’t any tradeoffs in your proposed solution? Not likely, it’s much more likely you’re missing something. You should have questions – nobody knows everything! If you don’t have questions, you’re missing something. Write all this stuff down too.

4. More input, better output.

You can’t connect things you don’t know about. Read in and around your space (not just your exact problem, also similar stuff so that you can let that simmer and let your brain make connections).

Find research papers in the same space. Even if you only understand a tiny fraction of the paper, it'll likely contribute to your ability to think about the problem. Look (critically!) at other solutions too. Great solutions can come from tearing apart someone else’s idea.

It's important to think about the kind of input you're consuming. Garbage in = Garbage out. Maintain a ridiculously high standard for the quality of content you're willing to consume. Your brain will make use of it regardless of whether it's good or not, so make sure it's good. We're blessed with abundance and we can greatly improve the quality of our software by being particular about the information we consume.

5. Tradeoffs

Everybody says design is about tradeoffs, but you need to look at at least two possible solutions and figure out what’s good and bad about those things before you can really say you made a tradeoff. Write this down too!

Before deciding on a course of action, come up with three alternatives. Here are three ways of thinking about The Rule Of Three.

One more formulation: If you don’t have three options for a solution to a problem, you don’t understand it well enough yet, and you might need to explore it more.
Family therapist Virginia Satir encouraged people to have at least three choices. She said: "…to have one choice is no choice; to have two choices is a dilemma; and to have three choices offers new possibilities. –The Satir Model
Jerry Weinberg (who came up with this rule’s name) puts it another way that should get your attention: If you can’t think of three things that might go wrong with your plans, then there’s something wrong with your thinking.

6. Extreme Focus

This is where the hammock comes in. One of the great things about the hammock is that if you lie down with your eyes closed, no one knows you’re not sleeping and they won’t bother you because you might be sleeping so you can focus completely on your problem without interruptions.

The computer is a prime source of distractions – you need to get away from the computer if you want to focus. Build focus time into your calendar. It won't just happen and you need to create that time and space if you want to do your best work.

You can’t do everything – if you really focus on something you’ll drop other balls like returning phone calls or emails, or remembering to run that errand or fix that bug you said would be really quick. Let your loved ones know what you’re doing and why you’re not going to be super responsive for a while.

Waking Mind vs Background Mind

Hickey has a bunch of interesting stuff to say about what he calls your waking mind and your background mind. He uses his waking mind to assign tasks to his background mind and to analyse its products.

Your background mind is where you solve most non-trivial problems but you can only feed it, not direct it. You need to spend enough time thinking about something for it to become an agenda item for your background mind to get anything useful from it.

A great analogy is that your waking mind is great at finding the local maximum, but it's not so great at figuring out that there's another hill further down the path that will take you even higher.

In his book Deep Work, Cal Newport talks about Unconscious Thought Theory, which is closely related to this. Unconscious thought theory attempts to understand the different roles of conscious and unconscious thought in decision making.

The theory states that if a decision is about strict rules, your conscious mind is better. If it's large, vague and has conflicting information - leave it to your unconscious mind. The reason it's better is that it has more neural bandwidth available, so it can sift through and understand everything a whole lot better, providing you with better solutions.

What should you do to facilitate this? Rest. Load up your background mind and then shutdown properly. When you give your conscious brain time to rest, you activate the unconscious brain.

To load up your background mind you review everything you wrote down in the previous steps and really focus on it. Because your waking mind can only keep track of so many pieces of information at once (7 +- 2, known as Miller's Magic number), you need to have written down everything you know about your problem so that you can swap those pieces back into your working memory when you need them. To load all of this into your background mind you need to survey everything you’ve written down to give your background mind the chance to make connections between different things that your working mind didn’t realise could be connected.

Hickey stresses that you can’t do this at the computer, you need to go somewhere and have no external input. He recommends going somewhere you can close your eyes but not go to sleep so you can sort of switch your brain from external to internal input(all that information about your problem that you just reviewed).

The other sub-step is to leave the problem alone for a while and let your background mind do its thing. Sleep is a great way to do this, or going for a walk, or anything that gets your waking mind off of the problem. If you’re working on a hard problem you should sleep on it for at least one night, more if it’s a really hard problem. Just because you have an idea right away doesn’t mean you won’t have a better one in the morning. And once you have an idea, analyse it, don’t just run with it. Just because you slept on it doesn’t mean that solution that popped into your head in the morning is perfect.

Lastly, but importantly, code is required. You need to take your great ideas and make sure that they solve the problems you think they solve.

Personal Takeaways

Hard problems take time to solve and if you're not spending time thinking hard about them, you're probably not solving them in the best way possible.
Thinking through a problem beforehand is how you grow more confident in implementing solutions you haven't tried before.
Most of the biggest problems in software are problems of misconception. It's not that they're that tough to solve, but more that we jump into it without giving it the required thought, and without understanding the problem properly.
Building software to solve a problem isn't difficult, identifying and understanding the problem is the difficult part. You cannot engineer well if you don't understand the problem you're solving. This is the value that domain-driven design brings to engineering teams. This ties in with the ideas from Fred Brooks' No Silver Bullet :

"I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the lab or of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.

Paper Club Discussion Notes

This talk was discussed as part of a paper reading club that I host at work. These are some of the points and reading recommendations that were raised during the session.

With regard to point 4 - More input, better output: This doesn't necessarily mean consuming possible solutions, but more so about consuming information that is going to help you understand the problem.
From Toyota Kata: If the solution isn't obvious you don't understand the problem well enough.
A related model is the MOIJ - Motivation, Organisation, Information, Jiggling - by Jerry Weinberg. The J is most closely related to the contents of this talk.

On the importance of writing:

We don't formulate ideas by thinking them to ourselves, but by expressing them to others. Write down your ideas because it's the fastest way to gain clarity and validation.
Tweet things. Forcing yourself to condense something into 280 characters is a great way to a) show that what you're thinking is actually garbage, or b) consolidate your thoughts and extract the core idea in it.

Cross-functional teams are not just about developers, data scientists and business analysts, but also require domain expertise.