On bottom up design and dependency hell

Posted: August 3, 2012 in Uncategorized
Tags: , ,

Bottom-up design is a powerful technique. Perhaps Paul Graham said it best:

In Lisp, you don’t just write your program down toward the language, you also build the language up toward your program. As you’re writing a program you may think “I wish Lisp had such-and-such an operator.” So you go and write it. Afterward you realize that using the new operator would simplify the design of another part of the program, and so on. Language and program evolve together. Like the border between two warring states, the boundary between language and program is drawn and redrawn, until eventually it comes to rest along the mountains and rivers, the natural frontiers of your problem. In the end your program will look as if the language had been designed for it.

I’ve personally experienced the power of bottom-up design in my work on my current hobby-project clisk, which is a library for procedural image generation. By building up a set of basic image generation operations, I’ve created a DSL which is proving to be extremely expressive for generating some quite sophisticated procedural images. As an example, the following code:

  (scale 0.4
    (let [z (v+ (v* 2.5 plasma) -0.75)
          colour (landscape-map z)
          height (v* 3.0 (vmax 0.5 z))]
      (render-lit colour height ))))

This generates a nice fractal landscape texture map with a bit of shading:


Note clisk isn’t a fractal landscape generator (the only thing that is landscape-specific is the hand-crafted colour gradient used for the landscape colours) – it’s just a general purpose procedural image generation DSL which has sufficient expressive power to generate fractal landscapes in just a few lines of code. This is the power of bottom-up design in a language like Clojure.

So what’s the downside?

The issue I encountered was that of managing dependencies. If you build bottom up, you are necessarily creating dependecies on the fundamental building blocks that form the foundation of your bottom-up design.

Dependencies aren’t necessarily a bad thing as long as they are stable. But that isn’t always the case. In clisk, for example, I discovered I needed some extra functionality which necessitated changing the fundamental data structure that I had used to represent the nodes in my DSL. Refactoring this was a *huge pain* which took perhaps a whole day of work because I had to rewrite and re-test 50-75% of the higher-level functions in the DSL that depended on the code node data structure. Not pretty.

But then I noticed something – my sample code using the DSL still worked (e.g. the landscape map generating code above). Despite all the refactoring pain I had gone through, none of the samples needed any refactoring. This was a lightbulb moment…..

So what is the solution to managaing dependencies in bottom up design?

Build against an abstraction

Observe the difference:

  • In my library code, I had built everything against a concrete data structure. When that changed, all the dependencies broke.
  • In my sample image code, I had built against an abstraction – the clisk DSL itself – which insulated me from the changes in the underlying data structure. When I rewrote the internals of clisk, everything still worked.

I think the principle applies in general to bottom up design. You should aspire to build in layers, each depending on an abstraction which is as simple as possible but no simpler to enable you to use the capabilities of the layer below.

In the clisk case I had three layers:

  • Sample code
  • Clisk DSL functions
  • Base data structure

But this probably needed an extra layer:

  • Sample code
  • Clisk DSL functions
  • Primitive operations (on data structure)
  • Base data structure

If I had done this in the first place, then the refactoring work would have been minimal as I would only need to rewrite the primitive operations – not the whole extensive library of Clisk DSL functions.

So this turned into a bit of ramble…. but I think I’ve learned something valuable and I hope it is useful in providing some insights into the pros and cons of bottom up design. And as often seems to be the case in Clojure, abstractions turn out to be the hero that saves the day…..

  1. Andreas says:

    that’s a great blog post full of trueness! The same insights you explain I’ve learned for myself in my side projects. Design and develop against *abstractions* (functions and such), *not* against a *concrete* whatever (e.g. data structures).

    • mikera7 says:

      Thanks! I think this is good advice for most programming languages, but somehow using Clojure seems to make it more conspicuous. Perhaps I got spoilt with refactoring Java IDEs beforehand (which let you get away with a multitude of sins).

  2. Vid the Kid says:

    Clisk is also a command-line alternative to Skype’s GUI. Thought you might like to know that.

  3. Hugo Estrada says:

    Great entry. Coming from OOP it almost seems hard to remember that while programming in clojure.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s