“There are only two hard things in Computer Science: cache invalidation and naming things (and off-by-one errors)”
Clojure is a great language, and one of the few things I dislike about coding in Clojure are namespaces.
It is not because namespaces are fundamentally a bad idea – in fact namespaces are a great feature and I believe they are fundamentally important if you are trying to manage a code base effectively.
But they are particularly fiddly to get right in Clojure. They cause a lot of unnecessary grief. They are not Simple. This is the story of my battle with the namespace demons to construct a decent namespace for Clisk live-coding.
The trouble with namespaces….
First of all, let’s get my rant out of the way. A few things I dislike about namespaces in Clojure:
- Namespaces complect naming things with the organisation of your source files. Often it is a good idea to keep this in sync (see Phil’s great comment below) however it is not clear that you *always* want to tie these together. It also introduces subtle complexities (e.g. the frequent confusion between hyphens and underscores – a namespace “my-ns” needs to be in a file “my_ns.clj”. Yuk.)
- Vars complect the concept of a storage location (as with other Clojure reference types such as atoms or refs) with participation in the namespace system, with lots of important metadata (e.g. arcane methods to specify if the var is a macro) and with dynamic binding techniques. It’s very clever… but far from being simple…..
- Local aliases are great: (alias/some-foo-from-another-namespace)…. right up until you need to copy/paste some code from a different namespace that uses a different alias name. Then you need to go and fix all the aliases. This problem doesn’t happen in Java: every class name must be either imported (so you can use it unqualified) or fully qualified (you can use it wherever you like)
- The (ns …) macro is cryptic and arcane. Someone at the Clojure Conj described their definition of a Clojure expert as “someone who can write a new ns declaration from scratch without copying from somewhere else or looking up the syntax”. Not a very beginner-friendly way to start each source file!
- Error messages in the loading of namespaces are often incomprehensible. This makes it very hard to track what is going wrong. Also, since Clojure is such a dynamic language you don’t get any real help from static namespace checking and only discover these problems at load/compile time.
- There is no easy way to import things transitively. e.g. suppose I want a clisk.live namespace to do some live coding, and I want it to import everything from clisk.functions, clisk.node, clisk.patterns, clisk.colours etc. I can make this namespace, but if I then do “(use ‘clisk.live)” from the REPL then I don’t get the imported functions…. aaargh!!
The namespace importing problem
Well, I can’t fix everything right now, but I can at least tackle the last issue: namely how to assemble a namespace with transitively importable functions, macros etc.
So the namespace I’d like to be able to use looks a bit like this (simplified):
(ns clisk.live (:use [clisk core node functions patterns colours util effects]))
i.e., for live coding we want to have all the key namespaces imported and ready to use at our fingertips, without having to type namespace aliases all the time. This isn’t particularly good practice for coding in general, but for live coding experimentation it is generally the case that speed and low conceptual overhead is more important than long-term maintainability.
This namespace works fine as defined, but there is a problem: when you use clisk.live from elsewhere, then the new namespace doesn’t import all the functions you happily imported into the clisk.live namespaces.
Why does this happen? The problem is with the refer functionality in Clojure (which is called indirectly by use and its variants. This refers to all the “public vars of a namespace”. Digging deeper, we find that ns-publics in turn defines these as any vars in the namespace mapping that have a namespace equal to their current namespace. So here’s the problem: the vars in the clisk.live namespace map don’t count as public because the vars themselves aren’t defined in the clisk.live namespace. So they don’t get picked up by use and friends.
So what are our options? I think there are two obvious ones:
- Hack some combination of use, refer or ns-publics to do what we want
- Make new vars in the clisk.live namespace that count as public
Option 1. looks messy and would probably break lots of other things. So we choose option 2.
Making it work
Luckily, I found some inspiration by seeing how others had tacked similar problems:
- The import-fn and import-macro from Zach Tellman’s Potemkin library: potemkin.namespace
- This stack overflow question and particularly the answer by John Aspden
By combining the ideas from these, I managed to build a little library of macros (mikera.cljutils.namespace) that do the namespace imports as required and create new var definitions in the current namespace. The code is a bit involved, mostly because of the need to handle metadata and various special cases.
So we can now do:
(ns clisk.live (:require [mikera.cljutils.namespace :as n])) (n/pull-all clisk.node) (n/pull-all clisk.functions) (n/pull-all clisk.patterns) ...
The pull-all macros simply scan the target namespace for all public vars, then create new vars in the current namespace with the same value and metadata. Because these new vars are public, they will be imported sucessfully whenever you use the “clisk.live” namespace, i.e. you can live code to your heart’s content using functions from all the different underlying namespaces, e.g.:
(use 'clisk.live) (show (offset (vsplasma vsplasma) (sunset-map spots)))
Which gives us an interesting flecked pattern, that might make a good polished stone texture:
Well, we solved the problem and can now do elegant live-coding in Clisk by using a single combined namespace, which is nice.
More generally, I think the whole namespace issue is still an area where Clojure could do with a lot of improvements, based on practical use cases. Some humble suggestions:
- Make the functionality I implemented above part of core Clojure or at least an approved contrib library (e.g. tools.namespace), so people don’t feel the need to reinvent the wheel.
- Revisit the whole ns macro for Clojure 2.0 and make something that is much simpler and cleaner. It’s a major hurdle, especially for beginners, and I think is worth a breaking change / full replacement.