Monads in Small Bites - Part III - Monoids

This is Part III of my Monads tutorial. Make sure you read the previous parts:

Monoids

Simply put, Monoids describe types containing a binary function and an identity value.

When applied to the identity value and a random value x, said function leaves its argument x untouched, returning it as a result.

This short description should be enough to get the conversation started.

Here’s how Haskell defines a Monoid:

1
2
3
4
5
class Monoid m where
    mempty :: m
    mappend :: m -> m -> m
    mconcat :: [m] -> m
    mconcat ms = foldr mappend mempty ms

This type introduces three new functions so let’s walk through each one of them:

mempty - I started with a lie since mempty isn’t actually a function. You can think of it as a constant of the same type of the Monoid m. It is this monoid’s identity value.

mappend - A poorly named function, mappend is the binary function I mentioned earlier. It receives two arguments of type m and returns a value of type m

mconcat - It receives a list of Monoids m and reduces them to a single Monoid of type m. What’s interesting about this snippet is that the Monoid type class provides a default implementation for mconcat: it simply calls foldr with the binary function mappend, a starting value of mempty and the list of Monoid values ms

Enough Haskell! Let’s have a look at a few examples.

Did you know that, in Clojure, the functions * and + are monoids? Yup. But don’t take my word for it. Let me prove it to you:

1
2
3
4
5
6
7
8
(def  mempty (+)) ;; 0
(def  mappend +)
(defn mconcat [ms]
    (reduce mappend mempty ms))

(mappend 3 4) ;; 7

(mconcat [2 3 4]) ;; 9

Whoa! What happened here? Am I just making this stuff up?

Not really. I only defined the same haskell names to their Clojure counterparts for clarity. Totally overkill. The code above is the same as:

1
2
3
4
5
(+) ;; 0

(+ 3 4) ;; 7

(reduce + [2 3 4]) ;; 9

Did you notice that on the second call to reduce we did not provide an initial value? That’s because reduce will attempt to get its initial accumulator by calling the reducing function without arguments - hence mempty == (+).

So that means we don’t even need an mconcat function since in Clojure, reduce works with monoids as well!

Update: this isn’t entirely true. When I wrote this post I had in mind the version of reduce provided by the Clojure (1.5+) reducers library. The source code shows how that is the case.

The implementation of reduce in clojure.core however uses the first element of the collection being reduced over as its seed.

But how the hell do you create a monoid in Clojure then? I’m glad you asked. Let’s create our own plus-monoid!

Your first monoid

In Part I I implemented Functors using protocols and records. In Part II I showed how Applicative Functors could be implemented using multimethods.

This time around I won’t be using any of these. I’ll implement Monoids using pure functions:

1
2
3
4
5
6
7
8
9
10
11
(defn plus-monoid
    ([]
        0)
    ([a b]
        (+ a b)))

(plus-monoid) ;; 0 - same as mempty

(plus-monoid 3 4) ;; 7 - same as mappend

(reduce plus-monoid [2 3 4]) ;; 9 - when working with monoids, reduce is the same as mconcat

We start by defining a function with multiple arities. The first body receives no arguments, so we just return the identity value for summation, which is 0 (zero). The second body receives two arguments so we can just add them up. Multiplication can be implemented in a similar fashion but obviously with the identity value of one.

Easy, huh?

Oh, by the way, lists are Monoids too! Who’d have thought?

Here’s its Clojure implementation:

1
2
3
4
5
6
7
8
9
10
11
(defn list-monoid
    ([]
        '())
    ([a b]
        (concat a b)))

(list-monoid) ;; () - remember, same as mempty

(list-monoid [1 2 3] [4 5 6]) ;; (1 2 3 4 5 6) - remember, same as mappend

(reduce list-monoid [[1 2 3] [4 5 6] [7 8 9]]) ;; (1 2 3 4 5 6 7 8 9) - mconcat in action

Same rules apply but for lists mappend is achieved by using concat inside our monoid function.

Also, since our binary function concatenates two lists together it makes sense that mempty is () (the empty list). Remember mempty is supposed to be an identity value so if we stitch () and [1 2 3] together, we’re left with [1 2 3] which is exactly what we’d expect.

You can see now why I said mappend was poorly named. While it makes sense when you think about lists, mappend doesn’t do any appending in our plus-monoid and in fact most monoids don’t append anything. Just keep this in mind if you see any haskell code using it: mappend is just a binary function.

Don’t break the law

You saw this coming, huh? Monoids also come with a couple of laws. You know the drill. Let’s prove they both hold.

Identity

Applying mappend to mempty and a monoid x should be the same as the original x monoid.

In Haskell:

1
2
mappend mempty x = x
mappend x mempty = x

And the proof in Clojure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
;; first, the plus-monoid
(def mempty (plus-monoid))

(def x 10)

;; This...
(plus-monoid mempty x) ;; 10

;; ...is the same as:
(plus-monoid x mempty) ;; 10

;;now, the list-monoid

(def mempty (list-monoid))
(def x [1 2 3])

;; This...
(list-monoid mempty x) ;; (1 2 3)

;; ...is the same as:
(list-monoid x mempty) ;; (1 2 3)

Associativity

Applying mappend to a monoid x and the result of applying mappend to the monoids y and z should be the same as first applying mappend to the monoids x and y and then applying mappend to the resulting monoid and the monoid z

In Haskell:

1
mappend x (mappend y z) = mappend (mappend x y) z

And the proof in Clojure - remember that calling the monoid function with two arguments is equivalent to mappend in haskell:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
;; first, the plus-monoid

(def x 10)
(def y 25)
(def z 40)

;; This...
(plus-monoid x (plus-monoid y z)) ;; 75

;; ...is the same as:
(plus-monoid (plus-monoid x y) z) ;; 75

;;now, the list-monoid

(def x [40])
(def y [10 25])
(def z [50])

;; This...
(list-monoid x (list-monoid y z)) ;; (40 10 25 50)

;; ...is the same as:
(list-monoid (list-monoid x y) z) ;; (40 10 25 50)

Almost there…

This puts an end to Part III. It’s time to head to the pub.

When you’re back look for the final post in these series - Part IV - where we will conclude our journey by finally introducing Monads!

Monads in Small Bites - Part II - Applicative Functors

This is Part II of my Monads tutorial. Make sure you read the previous parts:

Applicative Functors

In Part I I talked a little about Haskell type signatures and introduced Functors, which provide a way to map standard functions over values which are wrapped inside a Functor - we used fmap for that. You might want to skim through it again as a refresher.

Now suppose you have Functors that wrap functions and that you want to apply those wrapped functions to other Functors, maybe even composing new functions on the way!

What then?

Well you’re in luck! Applicative Functors do just that! They’re Functors on steroids.

Here’s how Haskell defines the Applicative data type:

1
2
3
class (Functor f) => Applicative f where
    pure :: a -> f a
    (<*>) :: f (a -> b) -> f a -> f b

Based on our previous knowledge of Haskell’s type signatures, we can infer from this definition that in order for it to be an Applicative Functor, f must already be a Functor.

Let’s break this down and have a closer look at the two new functions this type introduces:

pure is a function that takes a value a and wraps it into a minimal Functor f.

<*> is a function that takes two arguments: the first is a Functor f that wraps a function of type a -> b. The second argument is a Functor f that wraps a value - which could be a function! - of type a. The final result is a Functor f that wraps some value of type b - which was obtained by somehow applying the function (a -> b) to the Functor f a.

pure has a straightforward explanation whereas <*> is a bit more involved.

To clear things up, I’ll show the type signatures again but this time as if they only worked with the List Functor that we’ve been working on:

1
2
pure :: a -> [a]
(<*>) :: [(a -> b)] -> [a] -> [b]

Let’s revisit those definitions:

pure is a function that takes a value a and puts it into an empty list, returning the resulting single element list.

<*> is a function that takes two arguments: the first is a list containing one or more functions of type a -> b. The second argument is a list of one or more values - or functions! - of type a. The final result is a list of one or more values of type b - which was obtained by somehow applying the function (a -> b) to the Functor f a.

Enough definitions though! Let’s extend our List Functor and make it an Applicative as well.

While we’ll still be using the List Functor we implemented in Part I, this time I’ll implement its Applicative version using multimethods for a change. Here’s the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
;; it dispatches on the record type since we could have implementations of pure for List, Maybe, Either etc...
(defmulti pure (fn [f _] f))
(defmethod pure List [_ v]
    "Wraps value v in a list"
    (List. [v]))

;; it dispatches on the class of the Functor instance passed in the 1st argument
(defmulti <*> (fn [fs _] (class fs)))
(defmethod <*> List [fs xs]
    "Unwraps the functions in fs, applies them to the Functors in xs, wrapping the result at the end"
    (List. (for [f (:wrapped fs)
                 x (:wrapped xs)]
                (f x))))

By focusing on the List as an Applicative Functor we can more easily understand what these functions do. From the code above, pure’s job is a simple one: all it does is wrap it’s argument v into a minimal List functor which in our case means a Functor wrapping a one element list.

<*> on the other hand is responsible for somehow unwrapping the functions brought in by the Applicatives in fs and applying them to the [Applicative] Functors in xs. It does that by using list comprehensions and wraps the result into a new List Functor.

Study this code carefully. It can be tricky.

Note: When I first encountered <*> I had no idea what this function was called. I asked the twittersphere and it seems it’s called apply. In the process of figuring this out I was enlightened by this conversation. It turns out <*> has several names. Can you guess which one is my favorite? :)

With the Applicative functions defined for our List, let’s take it for a spin:

1
2
3
4
5
6
7
8
(def fs (List. [#(* 2 %) #(+ 10 %)]))
(def xs (List. [1 2 3]))

(<*> fs xs) ;; List{:wrapped (2 4 6 11 12 13)}


(def g (pure List #(* 50 %)))
(<*> g xs) ;; List{:wrapped (50 100 150)}

There should have been no surprises here. Read the code again and make sure it’s all fresh before moving along.

Don’t break the law

Just as Functors, Applicative Functors also need to obey some laws:

Identity

Feeding a function f to pure and applying the resulting Applicative to the Functor v should be the same as directly mapping f over the Functor v

In Haskell speak:

1
pure f <*> v = fmap f v

And this is the proof, in Clojure:

1
2
3
4
5
6
7
8
9
(def f #(+ 2 %))
(def v (List. [10]))

;; given the above, this...
(<*> (pure List f) v) ;; List{:wrapped (12)}


;; ...is the same as:
(fmap v f) ;; List{:wrapped (12)}

Composition

The result of applying an Applicative Functor that yields the function composition operator to the Applicative u, then apply the resulting Functor to v and finally applying that result to the final Applicative w should be the same as applying v to w and then applying u to the resulting Applicative.

That was a mouthful! Let’s see how Haskell tells this story:

In Haskell:

1
pure (.) <*> u <*> v <*> w = u <*> (v <*> w)

I needed to cheat a bit in Clojure to prove this law since functions are not curried by default like they are in Haskell. But this code should still clearly show how this law holds:

1
2
3
4
5
6
7
8
9
10
11
12
(def u (List. [#(* 2 %)]))
(def v (List. [#(+ 10 %)]))
(def w (List. [1 2 3]))

;; Given the above, this...
(-> (pure List (fn [x] (partial comp x)))
  (<*> u)
  (<*> v)
  (<*> w)) ;; List{:wrapped (22 24 26)}

;; ...is the same as:
(<*> u (<*> v w)) ;; List{:wrapped (22 24 26)}

Homomorphism

The result of applying the pure value of f to the pure value of x should be the same as applying f directly to x and then feeding that into pure.

In Haskell:

1
pure f <*> pure x = pure (f x)

And in Clojure:

1
2
3
4
5
6
7
8
9
(def f #(* 2 %))
(def x 10)

;; given the above, this...
(-> (pure List f)
  (<*> (pure List x))) ;; List{:wrapped (20)}

;; ...is the same as:
(pure List (f x)) ;; List{:wrapped (20)}

Interchange

The result of applying an Applicative Functor u to the pure value of y should be the same as taking the Applicative obtained by calling pure with a function that applies its argument to y and then applying that to u

In Haskell:

1
u <*> pure y = pure ($ y) <*> u

This type signature presents new syntax so before proving the law in Clojure, I want to explain what ($ y) means.

In Haskell, $ is the function application operator. So if we give y a value of 10, I can show you that in this law $ essentially translates to a single argument function that applies its argument to 10:

1
2
3
4
5
6
7
8
let double a = a * 2 -- helper function. it doubles its argument

-- given the above, this...
(($ 10) double) -- 20

-- ...is the same as:
let dollarTen = (\a -> (a 10))  -- this is Haskell's lambda syntax. It's equivalent to ($ 10)
((dollarTen) double) -- 20

Now, to the proof in Clojure:

1
2
3
4
5
6
7
8
9
(def u (pure List #(+ 10 %)))
(def y 50)

;; given the above, this...
(<*> u (pure List y)) ;; List{:wrapped (60)}

;; ...is the same as:
(def dollar-y #(% y)) ;; it's called dollar-y to show the correlation with the explanation above
(<*> (pure List dollar-y) u) ;; List{:wrapped (60)}

This brings us to the end or Part II. Two down and two to go.

I hope you’re still with me but go home now.

Or better yet go to the gym lift some weights and think about these Functors on steroids. When you’re back, look out for Part III.

Monads in Small Bites - Part I - Functors

Today I join the already bloated group of people who wrote monad tutorials. It’s a bit of a ritual, really.

Different than most tutorials though I aim to take a different approach. The good news is that I won’t be comparing monads to burritos :)

People say one needs to have his/her own epiphany in order to understand Monads and reading explanations from others is of little help. My goal is to disprove that.

To that end, this tutorial will be split in four parts:

You might want to bookmark this page - once the other parts are up, I’ll update the list above with the links to them.

Before we start

I know what you’re thinking: Do I really need to know Applicative Functors just to grasp Monads?

Well, no. However, I found that gradually building your knowledge from Parts I, II and III will allow you to fully grasp monads without the need for burritos or elephants.

You should also be familiar with a functional programming language. Any language should be fine but you’ll get the most out of this tutorial if you’re familiar with Haskell and/or Clojure.

If you’re not familiar with Clojure, fear not - Clojure is a small language and the code snippets should still make sense if you put your mind to it - they’re all short and sweet. I also encourage you to re-implement the examples in your language of choice to gain a deeper understanding on the subject.

Ready then? Let’s dive in.

Just enough Haskell

This is not a Haskell tutorial but trust me when I tell you that learning just enough about its type signatures will make all the difference in the world in understanding the concepts I’m about to present.

Although I’ll be using a little bit of Haskell syntax, I’ll also provide implementations in Clojure. They are in order so you should be able to paste all code sample in the REPL and follow along if you wish.

Type signatures

I’ll make this quick. Say you have a function called mk-vec that creates a 2D vector with x and y coordinates. Such function could easily be coded in Clojure like this:

1
2
3
4
5
6
(defn mk-vec [x y]
  {:x x :y y})

;; using it
(def my-vec (mk-vec 10 15))
(:x my-vec) ;; 10

As you can see, this function takes two arguments - x and y - and returns a map - or hash if you come from Ruby - that wraps those values in it, providing an easy way to retrieve them.

Now if this function had been implemented in haskell this would be its type signature - read the comments:

1
2
3
4
5
6
7
8
9
10
import qualified Data.Map as Map -- just giving Data.Map an alias
mkVec :: a -> a -> Map.Map [Char] a -- this is the type signature

mkVec x y = Map.fromList [("x", x), ("y", y)] -- this is the implementation. You can ignore this part.

-- using it
myVec = mkVec 10 15
Map.lookup "x" myVec -- Just 10 
--
-- ignore the actual value that is returned. Read it as 10 for now.

Whatever comes after the :: is part of the function type signature. Read it like this:

This is a function that receives two arguments of type a - which means any type - and returns a Map of key/value pairs where the key is of type [Char] - technically a list of Char values but for all effects and purposes just read it as String - and the value is of type a - the same type as its arguments.

It might help to see the type signature in this light:

1
mkVec :: a -> a -> (Map.Map [Char] a)

It highlights the return value in parenthesis.

Now let’s have a look at a built in function, *. As you know, it performs multiplication and this is its type signature:

1
(*) :: Num a => a -> a -> a

Can you guess now what it means? I’m sure you can. There’s one small difference though: this function has a type constraint - it’s that Num a => … part of the signature. Read it like this:

This is a function that receives two arguments of type a and returns a value of the same type, as long as the type of a is an instance of Num

And that’s it for now. Read it again to make sure it’s fresh in your mind and then continue.

As we encounter more type signatures, I’ll walk you through each one of them - but if you got it up until now, you’ll easily grasp the other type signatures.

Functors

In a nutshell Functors are things that can be mapped over.

Haskell defines Functors like this:

1
2
class Functor f where
  fmap :: (a -> b) -> f a -> f b

Let’s dissect that type signature:

fmap is a function that receives two arguments: the first is a function that receives an argument of type a and returns a value of type b and the second is a Functor that contains a value of type a - represented by f a. The result of calling fmap is a Functor of same type - f - containing a value of type b, which is the result of applying the function to a.

Too much? Let’s have a look at an example: the List Functor.

If we rewrite the fmap type signature as if it only worked on lists, this is what we’d come up with:

1
  fmap :: (a -> b) -> [a] -> [b]

This new type signature allows us to rewrite that last definition:

fmap is a function that receives two arguments: the first is a function that receives an argument of type a and returns a value of type b and the second is a List of zero or more values of type a. The result of calling fmap is a List of zero or more values of type b, each of which is the result of applying the function to each a element.

Does this sound familiar to you? It should, because this is essentially what the map function available in most functional-ish languages does! It takes a function and a list, applies the function to every element in the list while putting the results into a new list, finally returning it.

In fact, the fmap implementation of the List Functor from Haskell is implemented in terms of map.

Let’s see how that could be done in Clojure. I’ll use protocols and records but they are not strictly required for this.

Here’s our Functor protocol:

1
2
(defprotocol Functor
    (fmap [functor f] "Maps fn over the functor f"))

And now our List Functor:

1
2
3
4
(defrecord List [wrapped]
    Functor
    (fmap [functor f]
        (List. (map f (:wrapped functor)))))

In the snippet above all we’re saying is that the List record must satisfy the Functor protocol, which makes sense. fmap then is responsible for unwrapping the value contained in the list functor and mapping f over it.

Give it a go!

1
2
3
(def my-list-functor (List. [1 2 3])) ;; List{:wrapped (1 2 3)}

(fmap my-list-functor #(* 2 %)) ;; List{:wrapped (2 4 6)}

We can now map arbitrary functions over the values wrapped in a Functor! Awesome!

Note: In our Clojure version of the List Functor, it is implemented as a Record that wraps a primitive Clojure list/vector - []. As I mentioned this is not necessary but I chose to do it here to explicitly show the relationship with the Haskell types.

Don’t break the law

Now that we got a feel for what Functors are, it’s worth noting that they must obey a few laws to be considered full-fledged Functors.

Identity

Mapping an identity function over a Functor is the same as applying identity to the Functor itself

This is how Haskell puts this law:

1
fmap id functor = id functor

Translating that to Clojure:

1
2
3
4
5
;;This...
(fmap my-list-functor identity) ;; List{:wrapped (1 2 3)}

;;is the same as:
(identity my-list-functor) ;; List{:wrapped [1 2 3]}

Composition

If you compose the functions f and g and map the resulting function over the Functor, that is the same as first mapping g over the Functor and then mapping f over the resulting Functor.

From the description you can see that this law involves function composition. In Clojure, that’s achieved with the comp function.

Again, let’s see how this law is defined in Haskell:

1
fmap (f . g) functor = fmap f (fmap g functor)

Note: The . operator denotes function composition in Haskell

And the proof in Clojure:

1
2
3
4
5
6
7
8
(def f #(+ 10 %))
(def g #(* 2 %))

;; given the above, this...
(fmap my-list-functor (comp  f g)) ;; List{:wrapped (12 14 16)}

;; is the same as:
(-> my-list-functor (fmap g) (fmap f)) ;; List{:wrapped (12 14 16)}

Note: make sure you’re familiar with Clojure’s threading macro: -> . I’ll be using it in most code snippets.

Nice! All laws hold so we can sleep peacefully in the knowledge that our Functor works as expected.

Now go get a drink and stay tuned for Part II, where I’ll introduce Applicative Functors.

This Is What We Built in 24 Hours

Last week our current client - ninemsn - ran the 3rd edition of their HackDay.

It’s akin to Atlassian’s ShipIt days so if you’re not familiar with the concept, you should check their page but here’s the gist:

  • We’re given 24 hours to work on whatever we like as long as it’s related somehow to the business

  • We’re also free to use any technology we want

The team consisted of @romainprieto, Pranav Raja, @thepaddedcell, @stewgleadow and myself (@leonardo_borges).

The idea: Cool story, bro!

@romainprieto came up with it:

  • A mobile app that could scan QR codes on a printed JIRA story card and provide features such as assigning a card to yourself, moving it between columns and adding comments to it.

This is a common need. Most people will manage their stories in some sort of software such as JIRA. But it’s also common to have a physical story wall.

The problem then is keeping them in sync. It’s all too common for people to move the cards on the well and forget all about JIRA.

Well, not anymore. This is exactly what our HackDay project tackles.

In 24 hours, we built two native mobile applications - Android and iPhone - that are able to scan QR Codes printed on the story cards and, through integrating with the JIRA REST API, provide all the necessary functionality needed for:

  • Assigning a story to yourself - whoever is logged in to the app
  • Moving the story between states - e.g.: In Analysis, Resolve Issue, Reopen Issue etc…
  • Adding comments to a given story

All code is open source and on github. Check it out! - and bear in mind this was written in 24 hours… ;)

We didn’t stop there though. We actually finished early and had a couple of hours to spare so we shot a video - yes, a video!!! - about the app. I’m telling you, this is Hollywood quality:

Also, checkout some screenshots of the Android version:

Enjoy! :)

Clojure, Leiningen 2 and Heroku: AOT Compilation Gotchas

Recently I upgraded the clojure project I’m working on to Leiningen 2 in order to start using nrepl - since swank-clojure is now deprecated. Little did I know this would lead me to a small debugging adventure.

Heroku

I use Heroku as my deployment platform and my project had been running on it for a few weeks without any issues. I also use Heroku’s PostgreSQL solution.

However, by upgrading to Leiningen 2, my project started throwing some weird exceptions during deployment - it couldn’t connect to my database any longer. Everything was fine on my local environment though.

Upon inspecting the deployment logs on Heroku, I realized that the Heroku Clojure Buildpack tries to perform Ahead of Time (AOT) compilation if it identifies the project is running Leiningen 2.x

This means that after receiving a git push Heroku runs this command:

1
lein with-profile production compile :all

At first this might seem harmless. But step back for a second and think about what this means for code that depends on environment variables.

Case in point: Lobos

Lobos is a clojure library for manipulating database schemas akin ActiveRecord migrations in Ruby land.

The real-world example on their github page recommends this code snippet for configuring a database connection:

1
2
3
4
5
6
7
8
9
10
11
(ns lobos.config
  (:use lobos.connectivity))

(def db
  {:classname "org.postgresql.Driver"
   :subprotocol "postgresql"
   :user "test"
   :password "test123"
   :subname "//localhost:5432/test"})

(open-global db)

More often than not this isn’t ideal. You’ll probably want to customise your database configuration based on environment variables. Something along these lines:

1
2
3
4
5
6
7
8
9
10
11
(ns lobos.config
  (:use lobos.connectivity))

(def db
     {:classname "org.postgresql.Driver"
      :subprotocol "postgresql"
      :user (System/getenv "DB_USER")
      :password (System/getenv "DB_PASSWORD")
      :subname (System/getenv "DATABASE_URL")})

(open-global db)

And you’re good to go - until Heroku decides to AOT compile your code.

Compilation

The Clojure compilation process caught me by surprise. As stated by Rich Hickey on this IRC log entry, “a side effect of compiling Clojure code is loading the namespaces in order to make macros and functions they use available”.

This means that during compilation, any top level function calls - such as (open-global db) from above - will get executed. The same applies to macros that expand to top-level function calls.

If said function call has no side effects, you’re probably ok.

The trouble with open-global though is that it effectively attempts a connection to the database!

BAM!

On Heroku the environment variables you rely on - such as DATABASE_URL - aren’t available on compile time meaning a connection will be attempted to an empty url. All sorts of badness.

After having understood why the compilation process was effectively executing top level function calls, I then wrapped that call to open-global in a function declaration:

1
2
(defn init []
  (open-global (db)))

This will prevent eager evaulation during compilation since all defn does is bind the given function body to a var.

Then, in the code that needs migrations to happen - such as in my midje tests setup - I initialize the database like this:

1
2
3
4
(background
 (before :contents (lobos.config/init))
 (before :facts (migrate)
         :after (rollback :all)))

I spent some time in the #clojure IRC channel on freenode and it is accepted that top level function calls should be avoided anyway - especially if you’re dealing with side effects.

I also applied the same principle to other initialization code I have such as the configuration for Korma.

Happy ending?

Not quite. After having spent quite some time to get to this point, this was the new error that was stealing my sleep:

1
2
3
4
5
6
7
8
9
Caused by: java.lang.NullPointerException
    at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:332)
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:987)
    at clojure.lang.Namespace.find(Namespace.java:188)
    at clojure.core$find_ns.invoke(core.clj:3659)
    at clojure.core$the_ns.invoke(core.clj:3691)
    at clojure.core$ns_name.invoke(core.clj:3698)
    at my_app.env__init.load(Unknown Source)
    at my_app.env__init.<clinit>(Unknown Source)

This isn’t an unknown error. It basically means you’re trying to find a namespace that doesn’t exist. Or does it?

Once again this only happened if the code got compiled Ahead Of Time. Puzzling.

A couple of debugging hours later and I tracked it down to be a problem with clj-logging-config - a logging configuration utility for clojure. This is the offending code:

1
2
3
4
5
(ns my-app.env
  (:use
    clojure.tools.logging
    clj-logging-config.log4j))
(set-logger!) ;; this call causes the exception

set-logger! is a macro that makes use of the *ns* var. *ns* contains the current namespace and, due to what I’m assuming to be special semantics regarding this particular var, it shouldn’t be unquoted within a macro - which is why it was failing in AOT mode.

I opened an issue in the clj-logging-config github page. I eventually forked the project and fixed the issue in this pull request. That was then merged last Friday.

My code now works perfectly - regardless of AOT compilation.

Final thoughts

This was a nice little journey and I’m actually glad I went through it. I have a much better understanding of Clojure’s compilation process, worked out some quirks on Heroku and even got my first Clojure related open-source contribution accepted. Life is good.

Hopefully this will save other people a lot of time and effort debugging similar issues.

Sean Corfield: Clojure and CFML Sitting in a Tree

Last night I attended the Adobe User Group here in Sydney. That might strike some of you as a big surprise given my relationship with Adobe is pretty much limited to fiddling with Photoshop/Lightroom to get my photos looking nice.

However the reason for which I attended the meetup is that Sean Corfield - a prolific member of the Clojure community - gave a presentation on how he introduced and migrated most of his backend code at World Singles from CFML to Clojure - hence my interest.

What follows is a mix of my own notes and what I could remember from the slides:

Going faster

The original platform was built in CFML between 2001 and 2008. It was essentially monolithic procedural code. It was rewritten in 2009 - when they introduced OO and a new version of CFML

Brief Stats

  • 3 million members
  • 1 million emails sent every day
  • 2k concurrent users 24x7, on average

Clojure and WorldSingles

  • Tried .NET - ran into all sorts of trouble. Didn’t work well in production.
  • Tried Scala - Memory leaks in the built-in actor library were a deal breaker. The type system also wasn’t a good cultural fit.
  • Evaluated Clojure in 2010 - seemed like a good language to squeeze performance out of lower level components
  • Clojure version released in production in 2011
    • Rewrote remaining Scala Code
    • 6.3k LOC - 1.5k Tests LOC
    • equivalent to 4x the CFML LOC
    • Clojure is essentially used in the model (as in MVC)

Why add Clojure?

  • It’s fast - compiles down to JVM bytecode (and it’s faster than CFML)
  • Immutability (automatic thread safety)
    • they found several thread safety bugs in the third party ColdFusion libraries being used
  • Built-in concurrency / parallelism support - future, pmap, pvalues etc.
  • Lazy sequences - being able to work with potentially ininite collections without bringing your server down.
  • All high quality, production ready Java libraries easily accessible via java interop.

What do they use it for?

  • Email
    • html generation & sending
    • tracking & log file analysis
  • Geolocation (rest/json)
  • i18n
  • Reporting (parallel queries) - he showed a bit of this code. Heavy use of futures + deref
  • Search engine integration (json/xml)
    • This breaks down to two Clojure components:
    • One feeds the search engine based on changes to the users profiles
    • The other then runs against the search engine 24x7 trying to find new matches for users
  • (lightweight?) Persistence layer
    • no ORM - thin framework over sql instead
    • Same interface for both Mysql & MongoDB
    • MongoDB being used after hitting performance issues in Mysql

Clojure ecosystem/community

Having an active Community is crucial for language adoption and the clojure community got this right:

  • Community
    • 6700 developers on the mailing list
    • ~300 developers on IRC 24x7
  • Active library development
    • #23 language on Github
    • Over 7k projects on github
    • Nearly 2k libs on Clojars

The Future

  • Looking into Cascalog for big data processing
  • All new back-end/model code written in Clojure
  • Views/Controllers will remain in CFML
    • This ties up with Sean’s answer to ‘What would you not use Clojure for?’
    • He mentioned HTML rendering as the major reason. You can’t hand hiccup code to a designer.
    • Similarly, while designers can work on Enlive templates, updating the dynamic bits in Clojure can be cumbersome.

Summary

It was great to hear the many good things Sean had to say about Clojure. WorldSingles seem pretty happy with the decision of migration their heavy-lifting code to it and hopefully these slide notes will give you some insight into the sort of things Clojure is good at.

Sean is likely to put his slides up somewhere so I’ll link to it as soon as I have it. Here they are.

If you want to know more about Clojure and be involved in the community, you should come to the next clj-syd meetup.

Build Automation With XCode 4.3, KIF and Jenkins

UPDATE: As pointed to me in this commment, sym links aren’t necessary for Waxsim anymore. Check Jonathan Penn’s fork on github.


After coming back from my holidays in China - which were awesome - I had no downtime at ThoughtWorks and started at a brand new client/project - a much needed change from what I had been working on lately.

It’s an iOS project, more specifically an iPad app and that’s pretty much all I can tell you about it. I will however tell you how my first few days in the project have been so far.

Being a brand new project I wanted to get our CI environment up and running as fast as possible - after all the project has only a couple of tests now. For context, here’s our setup:

  • XCode 4.3
  • GHUnit (I’m not gonna talk too much about it since this was the easy part - their docs are pretty decent)
  • KIF (for UI tests)
  • Jenkins
  • WaxSim (Hack to get the iPhone Simulator to run on the command line)

In most environments I had worked so far - Java, Ruby, Clojure etc.. - running acceptance tests from the command line is a given and so is integrating that into your build system.

Well, that’s just not the case with iOS projects. These tasks can be a real pain and take a while to complete. Here’s what I had to do:

On your development Mac

I’m assuming here you have an XCode project with at least one KIF test and that it builds successfully from the GUI, when you hit play. Now we can move on to run the tests from the command line.

This is the script I’m using to run my KIF tests. I call it RunKIF.sh and will walk you through it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/bin/bash

killall "iPhone Simulator"
rm /tmp/KIF-*

echo "Build the "UI Tests" target to run in the simulator"
xcodebuild -target "UI Tests" -configuration Release -sdk iphonesimulator clean build

OUT_FILE=/tmp/KIF-$$.out
# Run the app we just built in the simulator and send its output to a file
# /path/to/MyApp.app should be the relative or absolute path to the application bundle that was built in the previous step
echo "Running KIF tests"
/tmp/waxsim -f "ipad" "build/Release-iphonesimulator/MyApp.app" > $OUT_FILE 2>&1

# WaxSim hides the return value from the app, so to determine success we search for a "no failures" line
grep -q "TESTING FINISHED: 0 failures" $OUT_FILE

success=`exec grep -c "TESTING FINISHED: 0 failures" $OUT_FILE`


# if there was a failure, show what waxsim was hiding and crucially return with a non-zero exit code
if [ "$success" = '0' ]
then 
    cat $OUT_FILE
    echo "==========================================="
    echo "GUI Tests failed"
    echo "==========================================="
    exit 1
else
    echo "==========================================="
    echo "GUI Tests passed"
    echo "==========================================="
fi

You’ll notice there are mainly two things happening in this script:

  • It runs xcodebuild to build the target against which your tests will run
  • uses WaxSim to finally run your tests

To get the building step to work, despite what KIF’s documentation DOESN’T tell you, you need to add KIF as a target dependency to your KIF Tests target. This SO thread can walk you through it.

Now on to the fun stuff! For some reason I didn’t have the time to dig into, WaxSim seems to expect XCode to be under /Developer/ on your Mac.

The problem is that since Apple started distributing Xcode through the AppStore, it now lives under /Applications/Xcode.app/Contents/Developer/. Bam! This can cause all sorts of issues when building both WaxSim and your project. So if you see weird error messages complaining about not being able to find DevToolsFoundation.framework and the like, these sym links will probably fix things for you:

1
2
3
4
5
6
7
8
sudo ln -s /Applications/Xcode.app/Contents/Developer/ /Developer
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsCore.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsCParsing.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsFoundation.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsInterface.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsKit.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsRemoteClient.framework /Developer/Library/PrivateFrameworks/
sudo ln -s /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsSupport.framework /Developer/Library/PrivateFrameworks/

I know, right? Moving on.

By now you should be able to successfully run RunKIF.sh and get meaningful results from your test. Let’s configure our CI server now, shall we?

On your CI Mac

Depending on which state your build box is, you might need to create the same sym links I created in the previous section. So ssh into your build box and try to build your project from the command line:

1
xcodebuild -target "UI Tests" -configuration Release -sdk iphonesimulator clean build

This time I’ll assume you already have Jenkins installed and running.

Your Jenkins server is probably running as the jenkins user. You need to change that. Jenkins has to be running within a Desktop session - that’s a requirement from WaxSim. It needs an active desktop session in order to launch the simulator - and thus needs to run as a user who can login to your CI Mac.

If you installed Jenkins using Homebrew, here’s a plist you can use to specify which user jenkins should run as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>Jenkins</string>
    <key>ProgramArguments</key>
    <array>
    <string>/usr/bin/java</string>
    <string>-jar</string>
    <string>/usr/local/Cellar/jenkins/1.439/lib/jenkins.war</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>UserName</key>
    <string>UserWhoCanLogintoThisMac</string>
</dict>
</plist>

Then load the new plist:

1
launchctl load -w  /Users/UserWhoCanLogintoThisMac/Library/LaunchAgents/org.jenkins-ci.plist

Don’t forget to enable automatic login on the CI Mac for the user Jenkins is using. Check this article to learn how to do it.

Now we’re mostly done. Just go ahead and configure your project.

Add a build step of the type Execute Shell and provide sh RunKIF.sh as the command string. Trigger your build and everything should work as expected.

Final thoughts

Once I actually got these steps together to write this post, it didn’t look like much. But the amount of time I wasted finding out what had to be done to actually make this work is just insane.

I’m shocked at how much behind iOS development tools seem to be. It just shouldn’t take this much effort and time only to get some UI tests automated. So far I’m disappointed with the tooling around the Apple ecosystem and I really hope this improves sooner rather than later.

Clj-syd Report #0

Last night we held the first ever meetup for the Sydney Clojure User Group!

When I decided to start running the meetups I had no idea we’d end up with 37 people on the first night! What a great turn out!

As for the content, here’s what you missed:

  • Running Clojure apps on Heroku - Lincoln Stoll (@lstoll)

Linc works for Heroku but ended up not spending a whole lot of time talking about that - instead he showed us how to build a simple web application using Compojure, a small Clojure web framework.

He then evolved the example by implementing it using Noir - yet another web framework that builds on top of Compojure, adding some helpful macros.

To top it off, the UI was implemented in ClojureScript - so that was all Clojure from end to end!

  • Paredit (in Eclipse) for the IDE junkie - Matt Quail (@spudbean)

If you’re into lisps and use emacs - if not you should anyway - you learned to love paredit.

However, not everyone gets to write Clojure for a living and a lot of us end up in some sort of IDE. If you’re in Java land, a popular choice is Eclipse and Matt showed us how you can get the paredit goodness right there using the Eclipse plugin counterclockwise.

Frustrated with the Clojure REPL? Wish you had more useful shortcuts? Decent line editing? Saving that nice function you’ve been working on right from the REPL? Then make sure you check out both the presentation and IClojure, Cosmin’s project.

Definetly worth a try.

Bayan was in one of our internal meetups. Some people raised the fact that we don’t see a lot of the design patterns made popular by the GoF book in languages such as Ruby, Python or Clojure and the overall opinion is that in those languages, some of the problems these patterns solve in, say, Java, simply don’t exist.

Bayan decided to dig up a few and show what’s the alternative in Clojure. Interesting perspective.

Want more?

Join us on Google Groups, submit a talk proposal on the wiki and RSVP to the next meetup!

See you next time!

Project Euler: Problem 4 in Clojure

I solved a few of the problems on Project Euler in the past, both in Java and Ruby, and thought it would be useful to redo them in Clojure, thus improving my skills on the language’s core functions and libraries. Today I’ll share problem 4.

Go ahead and read it but here’s the meat of it:

“Find the largest palindrome made from the product of two 3-digit numbers.”

From this statement we can tell two things: (1) we’ll need a function that can tell whether a number is a palindrome or not and (2) that the largest palindrome is given by the product of two numbers between 100 and 999, inclusive.

Let’s tackle number one first:

1
2
3
(defn palindrome? [n]
  (= (->> n str reverse (apply str))
     (str n)))

With our utility function in hand, one possible solution might be as follows:

1
2
3
4
5
6
7
8
(defn largest-palindrome []
  (apply max (filter #(palindrome? %)
                     (for [x (range 100 (inc 999))
                           y (range 100 (inc 999))]
                       (* x y)))))
(time
 (largest-palindrome))
;;"Elapsed time: 1405.358 msecs

While this works, I wasn’t happy with a couple of things in this solution. First, I thought I could do without using filter. Second, we have unnecessary multiplications going on, leading to poor performance - it takes ~1.4secs to finish.

You see, when we begin multiplying the numbers, we’ll see multiplications such as 100 * 100, 100 * 101, 100 * 102 … and then again, after the first loop is exhausted, 102 * 100, 102 * 101, 102 * 102 …

That led me to take a closer look at for, Clojure’s list comprehension macro. It’s a very powerful construct, providing 3 useful modifiers: let, while and when.

With that in mind, I refactored my first solution to look like this:

1
2
3
4
5
6
7
8
9
10
11
(defn largest-palindrome-1 []
  (apply max (for [x (range 100 1000)
                   y (range 100 1000)
                   :while (>= x y)
                   :let [z (* x y)]
                   :when (palindrome? z)]
               z)))

(time
 (largest-palindrome-1))
;;"Elapsed time: 689.262 msecs"

Here, the while modifier makes sure we aren’t wasting any time with unnecessary multiplications. The when modifier lets us get rid of the outer filter call. And as you can see, the solution is about twice as fast as its first version.

On top of that, it’s still pretty concise. Not bad.

Backlog: Ola Bini on Clojure/conj

As I promised in my previous post, here are the highlights of what Ola Bini had to say about Clojure/conj.

Clojure/conj is the largest gathering of clojure programmers, hackers and enthusiasts, with a single track of talks spanning three days. Ola selected his favorite talks and summarised each one of them.

The actual slides for each one of the talks can be found on github.

You can find Ola’s slides here.

Fun Data Structures

This is certainly a very interesting topic. The highlight here is the approach Clojure takes to handling immutable data structures efficiently: persistent data structures through structural sharing - maybe a good short talk for the next meetup?

However, even though efficient, there are cases where the space complexity of known algorithms gets compromised. More specifically, algorithms that rely on the ability of changing data structures in-place - such as the QuickSort - will suffer from this approach.

I asked Ola about it and the short answer is that you should use a different algorithm that takes advantages of trees instead, since they often don’t rely on in-place changing.

The long answer is a book called Purely Functional Data Structures - another one added to my wish list.

Logic/constraint programming

This is a topic that’s completely new to me and it sounds fascinating at first. It also ended up being the topic of the rest of the tech night, with Ola, Sarah and Jo - also ThoughtWorkers - giving us a Logic Programming 101 crash course using Prolog. Lots to read up on.

In the Clojure world, logic programming can be achieved via core.logic.

The book The Reasoned Schemer was also mentioned as a good read on the subject.

Cascalog (github repo)

Haven’t had the time to play with it yet but it seems extremely useful. It’s a Clojure-based DSL for processing “Big Data” on top of Hadoop.

There you have it. Come join us for our next meeting!