Coda and I have been using Clojure to build Skyliner for the last fourteen months or so. I thought it might be a good idea to write down some of our experiences with this, for the benefit of others considering it for practical work.
Learning languages is easy, learning the idioms is less easy
“Lisp has no syntax,” or so they say. It does have some, but significantly less than other languages. Clojure has a slightly larger pile of stuff that you could mistake for syntax, but, it’s still compact and simple. The tricky part isn’t the language so much as it is the slang.
As a seasoned engineer who theoretically “knows” a few dozen languages, I got productive with Clojure pretty fast. Nevertheless I definitely emitted some crappy code in my first few months. Stuff like:
(every? #(= % “success”) (map :status (:state task)))
Which I’d write like this today:
(->> task
:state
(map :status)
(every? #(= % “success”)))
Threading macros and transducers specifically took a few months to become second nature.
This is the kind of thing that would matter to you if you were going to try to onboard a few new engineers a week. I never read a tutorial, because this is a startup, and I did not have time. You’d probably want to rectify that mistake and review their stuff for a while.
When the going gets tough, the tough use maps
If I were going to give you a quick summary of what our codebase is like, I’d say it’s procedural code that manipulates maps. That is literally 90% of it. This is a lot less bad than it probably sounds if you’ve never written Clojure, because the entire language is oriented around manipulating maps and lists.
We keep the wheels on a few ways.
- Schemas for our maps are pretty handy, particularly when they’re of the user-supplied data variety. We’re using prismatic/schema for this, although if we were starting today we might use clojure.spec.
- Our codebase has better test coverage than nearly anything I’ve ever worked on.
- We use Kibit and Eastwood in our build pipeline for the sake of general cleanliness.
Bells and whistles are very rare
I kind of assumed writing Clojure professionally would involve communing with the grand harmony of the spheres, or something, but it really doesn’t. And this isn’t bad. It is actually extremely good.
In fourteen months I count about six uses of recur. I think I wrote some code using trampoline once or twice and then decided against shipping it.
We’ve written defmacro ourselves less than ten times. Most of those are for logging, so that we can grab the caller’s value of *ns*. Others are setting dynamically scoped variables for the sake of implementing feature flags. They’re all really simple macros.
Types of any kind are rare to a degree that astonish me. We’ve written a handful of protocols, for example our scm protocol is there to provide a uniform interface for both GitHub and private git repos. We have records representing different kinds of CloudFormation stacks that we create and manipulate. That is pretty much it.
Multimethods are less rare
One thing we do use more extensively are multimethods. We use this to dispatch asynchronous jobs in our workers, to fan out to different handlers from webhook endpoints, and to make transitions in our deploy finite state machine.
In other languages we’d probably want to use some object abstraction or other, but multimethods handle things like this cleanly.
Clojure is not Scala
I had some anxiety when we were getting started with Clojure, and that was grounded in my years of experience with Scala. Scala has scarred both of us for a number of reasons. Scala builds on JVM typing to erect additional complexity, and in my opinion the results are mixed.
Clojure doesn’t ask you to type anything if you don’t want to. That has its pluses and minuses, but you can write most of your code without getting into any slapfights with the JVM. So as a higher-level abstraction over Java, it works.
Building a server application with Clojure is a better experience than with many compiled languages, because as with any Lisp, you can just hotpatch everything in the REPL as you build it.
I’ll grant you that maybe Scala has answers to all of these problems now, as I haven’t had the pleasure of using it in several versions. Do not @ me to talk about this.
Nesting sucks
Although Common Lisp has return-from, Clojure has no facility like return or goto. This isn’t something you miss writing idiomatic Clojure, but sometimes you find yourself boxed into writing non-idiomatic Clojure. A good example of such a situation is dealing with a morass of heterogenous functions that can return error codes.
Let’s say that you have a list of steps that need to complete in a specific order, and may fail. Conceptually, in Python:
x = foo()
if not x:
return False
y = bar(x)
if not y:
return False
return baz(y)
This can be elegantly handled if the methods in the pipeline all return nil in the failure case, and we don’t care to do much else.
(some-> (foo) bar baz)
But things start to fall apart as the signatures of the functions in the pipeline vary, or if we want to instrument the pieces with logging.
(if-let [x (foo)]
(if-let [y (bar x)]
(if-let [z (goo x y)]
(do
(qux x y z)
(log "it worked")
true)
(do
(log "goo failed")
false))
(do
(log "bar failed")
false))
(do
(log "foo failed")
false))
We have a decent amount of old code that looks like this. It’s all well tested and in that sense it’s relatively safe, but it’s still craptacular and tricky to modify.
Before a throng of enlightened individuals amble up to the mic stand in the aisle to tell us this, I should say that we are wonk as hell and therefore realized we were building a composition of either monads.
But a highbrow-yet-idiomatic solution to that in a language otherwise devoid of category theory wasn’t immediately obvious. I messed around the idea of tackling this with specialized macros, but decided this was an unmaintainable tarpit.
In the end we decided to just try using a category theory library, cats. That lets you write something equivalent to the above like so:
(require '[cats.core :as m])
(require '[cats.monad.either :as either])
@(m/mlet [x (if-let [v (foo)]
(either/right v)
(either/left))
y (if-let [v (bar x)]
(either/right v)
(either/left))
z (if-let [v (goo x y)]
(either/right v)
(either/left))]
(m/return (qux x y z)))
Which cuts out the nesting and makes a big difference in sufficiently complicated scenarios.
It is unclear to me if the category theory would still be a win on a less experienced team. I have a long history of being skeptical of things like this, but it has improved our lives recently.
Thanks for reading!
I hope this helps if you’re considering building something real with Clojure.