Constructing a live-coding namespace

Posted: December 20, 2012 in clisk, coding, Uncategorized
Tags: , , ,

“There are only two hard things in Computer Science: cache invalidation and naming things (and off-by-one errors)”

Clojure is a great language, and one of the few things I dislike about coding in Clojure are namespaces.

It is not because namespaces are fundamentally a bad idea – in fact namespaces are a great feature and I believe they are fundamentally important if you are trying to manage a code base effectively.

But they are particularly fiddly to get right in Clojure. They cause a lot of unnecessary grief. They are not Simple. This is the story of my battle with the namespace demons to construct a decent namespace for Clisk live-coding.

The trouble with namespaces….

First of all, let’s get my rant out of the way. A few things I dislike about namespaces in Clojure:

  • Namespaces complect naming things with the organisation of your source files. Often it is a good idea to keep this in sync (see Phil’s great comment below) however it is not clear that you *always* want to tie these together. It also introduces subtle complexities (e.g. the frequent confusion between hyphens and underscores – a namespace “my-ns” needs to be in a file “my_ns.clj”. Yuk.)
  • Vars complect the concept of a storage location (as with other Clojure reference types such as atoms or refs) with participation in the namespace system, with lots of important metadata (e.g. arcane methods to specify if the var is a macro) and with dynamic binding techniques. It’s very clever… but far from being simple…..
  • Local aliases are great: (alias/some-foo-from-another-namespace)…. right up until you need to copy/paste some code from a different namespace that uses a different alias name. Then you need to go and fix all the aliases. This problem doesn’t happen in Java: every class name must be either imported (so you can use it unqualified) or fully qualified (you can use it wherever you like)
  • The (ns …) macro is cryptic and arcane. Someone at the Clojure Conj described their definition of a Clojure expert as “someone who can write a new ns declaration from scratch without copying from somewhere else or looking up the syntax”. Not a very beginner-friendly way to start each source file!
  • Error messages in the loading of namespaces are often incomprehensible. This makes it very hard to track what is going wrong. Also, since Clojure is such a dynamic language you don’t get any real help from static namespace checking and only discover these problems at load/compile time.
  • There is no easy way to import things transitively. e.g. suppose I want a namespace to do some live coding, and I want it to import everything from clisk.functions, clisk.node, clisk.patterns, clisk.colours etc. I can make this namespace, but if I then do “(use ‘” from the REPL then I don’t get the imported functions…. aaargh!!

The namespace importing problem

Well, I can’t fix everything right now, but I can at least tackle the last issue: namely how to assemble a namespace with transitively importable functions, macros etc.

So the namespace I’d like to be able to use looks a bit like this (simplified):

  (:use [clisk core node functions patterns colours util effects]))

i.e., for live coding we want to have all the key namespaces imported and ready to use at our fingertips, without having to type namespace aliases all the time. This isn’t particularly good practice for coding in general, but for live coding experimentation it is generally the case that speed and low conceptual overhead is more important than long-term maintainability.

This namespace works fine as defined, but there is a problem: when you use from elsewhere, then the new namespace doesn’t import all the functions you happily imported into the namespaces.

Why does this happen? The problem is with the refer functionality in Clojure (which is called indirectly by use and its variants. This refers to all the “public vars of a namespace”. Digging deeper, we find that ns-publics in turn defines these as any vars in the namespace mapping that have a namespace equal to their current namespace. So here’s the problem: the vars in the namespace map don’t count as public because the vars themselves aren’t defined in the namespace. So they don’t get picked up by use and friends.

So what are our options? I think there are two obvious ones:

  1. Hack some combination of use, refer or ns-publics to do what we want
  2. Make new vars in the namespace that count as public

Option 1. looks messy and would probably break lots of other things. So we choose option 2.

Making it work

Luckily, I found some inspiration by seeing how others had tacked similar problems:

By combining the ideas from these, I managed to build a little library of macros (mikera.cljutils.namespace) that do the namespace imports as required and create new var definitions in the current namespace. The code is a bit involved, mostly because of the need to handle metadata and various special cases.

So we can now do:

  (:require [mikera.cljutils.namespace :as n]))

(n/pull-all clisk.node)
(n/pull-all clisk.functions)
(n/pull-all clisk.patterns)

The pull-all macros simply scan the target namespace for all public vars, then create new vars in the current namespace with the same value and metadata. Because these new vars are public, they will be imported sucessfully whenever you use the “” namespace, i.e. you can live code to your heart’s content using functions from all the different underlying namespaces, e.g.:

(use '

(show (offset (vsplasma vsplasma) (sunset-map spots)))

Which gives us an interesting flecked pattern, that might make a good polished stone texture:

2012-12-20 Flecked pattern

Closing thoughts

Well, we solved the problem and can now do elegant live-coding in Clisk by using a single combined namespace, which is nice.

More generally, I think the whole namespace issue is still an area where Clojure could do with a lot of improvements, based on practical use cases. Some humble suggestions:

  • Make the functionality I implemented above part of core Clojure or at least an approved contrib library (e.g. tools.namespace), so people don’t feel the need to reinvent the wheel.
  • Revisit the whole ns macro for Clojure 2.0 and make something that is much simpler and cleaner. It’s a major hurdle, especially for beginners, and I think is worth a breaking change / full replacement.
  1. Phil says:

    You have some good points about the implementation of the ns macro being pretty lousy, but I strongly disagree that having to make your source files match the namespace files is a bad thing. It’s a minor bit of flexibility you give up as an author, but the regularity you gain as a reader is very valuable when exploring a codebase; no more asking yourself “I wonder where this is defined?”–it’s just obvious by looking.

    The fact that the compiler doesn’t even warn you when you have dashes rather than underscores is inexcusably stupid though.

    As far as vars as a storage location, this should be considered an implementation detail required to support interactive development. Without it you get into ML-style issues where redefinitions cannot be applied retroactively. (And in fact, in places like defrecord and deftype where you are forced to forgo the advantages of vars, this is exactly what happens, and it’s a huge mess.) But if application code is treating vars as storage locations, something deeply wrong is going on.

    The ns macro has gotten simpler since :use was deprecated, but it still does way too much and has awful error handling.

    • mikera7 says:

      Hi Phil actually I totally agree with you that you *normally* want to tie organisation of files to namespaces. The problems occur when the namespaces get big (e.g. 5,000 plus lines). You can’t easily break them up into logical units thanks to the restrictions on circular dependencies and the comparative lack of refactoring tools. I guess you can use “load-file”… but that always feels like a hack and could break all kinds of tool assumptions.

      • One could use load-file for solving the original problem exposed in your article as well. Or an hypothetical, more refined version that took a ns name as an argument rather than a string, and loaded the non-ns forms at most once (for switching fearlessly back and forth between namespaces).

        I fail to see any disadvantages in such an approach.

      • mikera7 says:

        Hi, vemv, I think there are *technically* no disadvantages to this, but we’re trying to solve for easy, transparent and understandable usage. I want namespace users to be able to just (use and have it work like any other namespace. It’s important from a human/social perspective to forcing users to learn yet another new namespace macro invocation. I guess it would be possible to reafactor things so that itself is constructed using load-file, but I think this would require changes in the source namespaces as well.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s