Composition over convention

If you’ve ever worked in a Ruby environment, and especially if you work with the excellent Ruby on Rails framework, then you will probably have seen the emphasis put on convention over configuration.

This philosophy has worked wonders in the Ruby world, but I’m going to argue that it is hitting the limits of it’s usefulness as we engineer more sophisticated applications that need to span across multiple frameworks and tools.

I believe that there is a more important principle that we should adopt. Rather than letting a framework handle everything for us by convention, we should be using higher order techniques from functional programming to explicitly compose our solutions from pluggable building blocks. Let’s call this principle composition over convention.

Some context

Convention over Configuration certainly works wonders in terms of getting you up and running quickly – making a fully functional Ruby on Rails application is quick and easy (in the Rich Hickey sense). For me it was a real breath of fresh air in comparison to the XML-based configuration madness that used to be required to build an application using some of the older Java “Enterprise” frameworks.

But while I enjoyed the productivity that this provided (and was frequently impressed by what people with far more expertise in RoR than myself could achieve), I always felt uneasy with the fundamental approach. It took a while for me to figure out exactly what was bothering me, but I think I eventually distilled it down to this point:

Frameworks don’t compose

A framework is a pre-built package, full of its own conventions, tools, and paradigms. You work within a framework – extending it by plugging in your functionality according to some well-defined extension mechanisms. You start with the conventional out-of-the-box defaults, then add or change things to make the framework do whatever you need. A sufficiently comprehensive framework with  a good plugin ecosystem (like RoR) can certainly take you a long way.

But suppose you need features from two different frameworks. You can’t add two frameworks together. All your plugins will be designed to work with one but not the other. The structure and design of your project that works naturally in one framework is likely to be fundamentally incompatible with the other framework. You are left with some unpalatable choices:

• Run the two frameworks side by side as two separate application stacks and write a lot of painful glue / integration. This is not fun, and is likely to give you a maintenance nightmare for the foreseeable future (especially if both frameworks continue to evolve separately)
• Port functionality from one into the other. This is a lot of work and risks reinventing the wheel. And taken to the extreme, this just means that one of your frameworks gets more and more bloated and complex over time until eventually it is neither simple nor easy.
• Give up. Sadly, I’ve seen this happen. It was a RoR project where the new functionality required (although clearly available in other frameworks/stacks) was too complex to re-implement and too challenging to integrate into RoR. Progress ground to a halt and the project was eventually cancelled.

Where did we go wrong?

Basically, we made a decision to build our application in a way that was tightly meshed with a framework that didn’t provide for the functionality we needed. That sounds obvious in hindsight but normally you don’t have the luxury of knowing 100% of your future requirements in advance.

I think we can do better than this.

My optimism stems from some of the emerging Clojure toolkits that are starting to demonstrate a way forward, beyond the world of convention-driven frameworks. These are based on composition first and foremost. Rather than starting with a framework and building out from it, you start with a toolkit of composable components and assemble them to create your finished application.

There seem to be some common features that are required to make a composition-based approach effective:

• Simple abstractions – a perfect example of this the central abstraction in Ring where a web server request/response handler is modelled as a function which takes a map and returns a map – nothing more. I can’t emphasize enough the importance of getting these abstractions right and making them simple – since implementations of the abstraction are the very things that you want to compose.
• Effective composition mechanisms – if you are going to assemble your complete application from composable components, it has to be both simple and easy. A good example is the middleware used in Noir (based on the Ring concept) which are actually higher order functions – so the composition mechanism is just standard Clojure function composition. I think that some form of “higher order” composition is essential – you need to be able to pass smaller components as parameters when you construct higher-level components. Consider for example that you have a component that generates a header for every web page – you need to have a way to pass this as a a parameter when you construct each of the components that generates a complete web page (if instead you store your header generator in a standard place where the page generators can pick it up and use it, then you’ve gone back the the “convention” school of design….)
• Layered bottom up design – I think this is almost essential for the composition-based approach to work. You have to start with simple composable primitives that implement an abstraction at one level (let’s say – a generator HTML snippets) and compose them into higher level components (e.g. a generator for a complete HTML page response) and compose these further into your ultimate solution (a complete web application).
• Users can choose to dive down where needed. For example, you might be happy with the default top level 404 page generator and compose this directly into your application. But for generating some custom HTML for your “like” buttons, you  might want to roll your own HTML snippet generator. It’s a bit like a Lego model kit that comes with most of the the big pieces already assembled. You can just plug them together and finish the model in a few seconds, or you can break them apart and put them together in a different way.
• Immutability – makes composition much, much easier. While it is possible to compose mutable things, getting it right is fiendishly trick. What are the concurrency issues? Do you need to take a defensive copy? Is there any monkey-patching going on that could break things? Immutable things can be both data or functions, but the important point is that they can be shared and composed safely without the risk of one part of the application affecting another (“spooky action at a distance”).  See also Rick Hickey’s recent talk “The Value of Values“
• Component libraries – are the way that you enable people to be quickly productive in a composition-based approach. Quickly plugging together some pre-assembled components is the only way that you will be able to compete with someone who is using an out-of-the-box framework like RoR. The components should implement the core abstractions mentioned above, and be designed so that they can be quickly composed together into a working solution.
• Wrapping functionality is needed to ensure that new features can be developed which participate as fully fledged, composable components.  Advanced users will almost certainly want to develop their own custom components. This includes wrapping external library functionality – indeed one of the great benefits working in Clojure is the huge ecosystem of Java libraries at your disposal. It also (very importantly) includes wrapping components from different composable stacks – this is how you solve the problem of bridging two different frameworks.

I’ve deliberately used the term “component” in a generic way. That is because I think there is quite a bit of flexibility in terms of what these actually are. In Ring, they are standard Clojure functions. In Clisk they are custom record data structures that drive code generation. In some DSLs they could be macros. In a rule engine that provides a separate interpreter they could even be pure data. I don’t think it matters as long as the abstraction they implement is well-defined and the composition mechanisms are sufficiently effective.

I’m personally very excited to see how the compositional style of development evolves in the Clojure world. While compositional approaches as outlined above are possible in any language there are some natural advantages that Clojure offers – it provides a language and developer culture where it is normal and easy to think in terms of immutability, higher order function composition and simple abstractions. As I have outlined above, I think these are three of the central principles for a composition-based development approach.

Advertisement