Enterprise Clojure Training

Clojure is a general-purpose programming language that combines interactive development with a robust infrastructure for multithreaded programming. Clojure is a dialect of Lisp that runs on the JVM and provides direct access to Java libraries. Clojure is a functional programming language that produces new values instead of changing objects. Clojure has been in the "adopt" category of the Thoughtworks tech radar since 2014 because it enables teams to build better software faster.

This course is divided into 12 bite sized sections: the Clojure ecosystem; Clojure syntax; functions; unit testing; control flow; functional programming; Java interop; concurrency; polymorphism; interacting with a database; specifications; and macros.

Each section has exercises to put the information into practise and re-enforce learnings.

2days, 10 hr/day + 2 hour webinar after completion of workshop.

At the end of this course, you will be able to

General programming knowledge.

Developers and Senior Developers.

The following software must be installed on your laptop prior to the course:

Complete the first 10 exercises on the 4Clojure website (http://www.4clojure.com).

If you would like to get a head-start, please read the official Clojure introduction tutorial (http://clojure-doc.org/articles/tutorials/introduction.html). This material will be covered as part of the course. Having read it before hand will allow you to focus on working through the exercises of the course.

A popular project built tool that provides a convenient way to pull libraries for your project. Follow the installation instructions at (https://leiningen.org).

As you can see, Leiningen created a project with one dependency; Clojure itself.

When you type in this code:

Clojure evaluates it immediately and returns a result:

Pressing the up arrow moves through your history.

The REPL is convenient for experimenting and doing informal tests. But the default REPL is not ideal for editing code.

Most popular editors have plugins to send commands from the editor to a REPL, do syntax highlighting and manage parenthesis. These are useful features, but I encourage you to prioritize learning Clojure ahead of configuring and learning new editor key combinations. It is difficult to do both at once!

For this course I recommend using IntelliJ (https://www.jetbrains.com/idea) with the Cursive Clojure plugin (https://cursive-ide.com). The main feature that sets Cursive apart is that it does error highlighting in the editor itself (https://cursive-ide.com/userguide).

Alternative: Lighttable (http://lighttable.com)

You can also open a REPL in your browser: (https://repl.it/languages/clojure).

For other editor options see (https://cb.codes/what-editor-ide-to-use-for-clojure).

Evaluate some math expressions in the REPL:

Create a new project called training. Open src/training/core.clj with your editor, write some expressions, and send them to the REPL:

Strings are enclosed in double quotes

Character literals are preceded by a backslash

Numbers can be Long

Double

BigInteger, suffixed with N

BigDecimal, suffixed with M

Expressed as exponents

Or ratio

Numbers are automatically promoted if they overflow during arithmetic.

Booleans are represented as true and false.

nil means nothing and is considered false in logical tests.

Lists are forms enclosed in parentheses.

Lists are evaluated as function calls.

The first element in the list is the function, and any following elements are arguments. Here we are calling the inc function on 1, which will return 2.

Quote yields the unevaluated form.

Apostrophe is a syntactic shortcut for quote.

Clojure has a sequence abstraction. Sequences can be lazy. Their values are only created as they are consumed.

Lists and sequences are printed the same way.

Symbols are resolved.

To create an unresolved symbol, quote it

Vectors are enclosed in square braces

Vectors have order 1 lookup by index and count. Vectors are used in preference to lists for cases where either could be used. Vectors do not require quoting and are visually distinct. You will rarely see or use lists.

Clojure compares by identity and by value. A vector with elements matching a sequence is equal to it.

Maps are key/value pairs

Maps have near constant time lookup by key. Maps are tuned to be fast. Maps are an excellent replacement for object fields.

Keywords are shorthand identifiers that do not need to be declared. Keywords begin with a colon.

Keywords are often used as keys in hashmaps; similar to fields in an object.

Keywords can be namespaced.

Double colon is shorthand for a fully qualified keyword in the current namespace.

Sets are written as

Sets have near constant time membership lookup, with a high branching factor.

Collections can be combined and nested

This is a map that has vector coordinates as keys and maps as values.

To call a function, wrap it in parenthesis:

The first element in a list is a function to be called. The remaining elements are the arguments to the function.

A var is used to store a mutable reference to a value. Vars are unbound if no value is supplied.

It is more common to supply an initial value.

Def created a var named x which is bound to the value 1. Vars are automatically dereferenced when evaluated.

To represent values that changes over time, you can use an atom.

We defined a to be an atom with initial value 1, then swapped the atom’s value with the inc function. We retrieved the value of the atom by dereference it with @. The current value of a is now 2, the increment of 1. @ is shorthand for deref.

Atoms provide compare and set, which is suitable for non-transactional changes. Refs provide transactional change, which is suitable for multi-threaded change management. Agents provide update serialization as an alternative strategy for multi-threaded change.

Deref also blocks and gets the result of futures, promises and delays, which are operations that do not block until dereferenced.

Symbols:

Symbols can be bound to a value in a scope with let.

The symbol x is bound to the value 1, and the function inc is called on x, resulting in 2.

The binding scope is within the parentheses enclosing the let form, and will shadow any existing bindings. It is preferable to use let instead of def for values that can be contained in a scope. Vars can be changed, but you should almost never modify them directly. Instead Clojure provides local bindings, atoms, refs and agents for managing change.

Destructuring is providing a literal data structure containing symbols that get bound to the respective parts of a value with a matching structure. Where we might otherwise bind the vector [1 2] to a single symbol, here we destructure two symbols x and y by providing a pattern that matches the vector.

Note that function arguments are already a destructured vector. The above case is an example of a vector of arguments which contains a vector of x and y.

Without destructuring we would extract substructure manually:

Destructuring can be used in any binding. We can use destructuring in for comprehensions:

In Clojure, for expressions are a convenient syntax alternative to map which also allows additional constraints to be expressed.

There is no need to restrict normalize to use 2 dimensions, instead we can write a generic version:

Variadic functions are destructured using &. Variadic means variable number of arguments. Arity means number of arguments.

Which produces a vector. Apply expands the vector arguments. Most mathematical functions are variadic:

Destructuring is nested, so you can use it to pull out sub-values without resorting to getter functions.

Common opportunities for destructuring are:

Values in a map:

Values in a sequence:

Nested destructuring

Namespace forms occur at the start of files.

The namespace must match the path and filename. The namespace training.core Must be defined in the src/training/core.clj file. Filename hyphens are replaced with underscores, and dot separators indicate directories.

The ns form allows us to require other namespaces and import java Classes. There are other valid ns forms which are best to be avoided and so are not shown here. If you do see them in other code, just know that you can and should achieve the same thing with the regular ns form described previously.

Clojure programs are written in expressions which are evaluated to results. If an expression needs to be compiled, it will be. Programs can be loaded from files or evaluated dynamically.

Regular expressions are written as #"pattern"

Write code into a new file called src/training/syntax.clj, and send the lines to the REPL as you enter them.

Note the prn and println behave slightly differently; prn keeps the quotes around strings. This is often useful when experimenting, because you can visually see the type of the values more clearly.

Note that the message global var is still the original value.

Functions are defined like this:

All functions return a result, the result of the last expression in the form. Defn binds the symbol square to a var which refers to a function which returns the result of multiplying the input parameter x by itself.

When evaluated, a list containing square in the first position causes the var bound to square to be automatically dereferenced to the function, which is called on the arguments.

Mathematical operators are regular functions which must be written in prefix notation.

Function arguments are evaluated from left to right before the function is called.

Unnamed functions are written as

Unnamed functions are also called anonymous functions and Lambda expressions. There is a special syntax for creating unnamed functions.

Is a function which increments a single argument.

Closures are functions that capture values from the environment.

Functions are values and can be passed as arguments to other functions. Functions that take a function as an argument are called higher order functions.

Map is function that calls a function on every element in a sequence

Map is a higher order function because the first argument is a function. Unnamed closures are useful as arguments to higher order functions.

Here we have the symbol x bound to 5. We call the map function. Our first argument is an unnamed function that captures x from the environment; a closure. The closure is called on every element of the vector 1 2 3, resulting in a sequence 6 7 8. Higher order functions, closures, and unnamed functions are terms that describe specific uses of functions that allow concise expressions.

You can make assertions about inputs and outputs of a function. Place a map after the arguments vector containing :pre and :post, which are a sequence of conditions which must hold true.

In practise pre and post are rarely used. It is more common to check for a condition and throw an exception:

Or to use a schema or spec (which will be covered later in the course).

While pre and post are more concise, they suffer the following drawbacks: Syntax is easy to get wrong, resulting in no assertion being made Assertions can be disabled Less control over error description and handling

We usually define functions with defn, which creates a global var to hold our function. But sometimes the function need not be globally available. We can specify functions without names like so:

But we would only do this if we wanted to make use of them in some way. The simplest way to use a function is to call it immediately:

The function appears as the first thing in a list, so is called on the argument 1, and evaluates the body of the function to calculate 2.

Another way to make use of an anonymous function is to bind it in a let form:

In Clojure it is very common to pass a function as the argument to another function:

So having a way to specify an anonymous functions is helpful:

You can name a function without creating a global var:

Naming a function has several benefits:

Note that

is shorthand for

There is a special syntax for creating anonymous functions concisely:

This allows the construction of very terse but powerful expressions:

I encourage you to use the (fn) form as much as possible instead of the #() form, it is not much more typing and affords more opportunity to name parameters and functions in meaningful ways which will describe your program better. For example:

Is longer, but provides a semantic summary of the operation and a hint at the expected input values.

An antipattern is

Prefer instead

Clojure supports keyword arguments, but this style is discouraged because it prevents users from passing a map of options. We cannot apply a map to a keyword argument function, so use a map argument instead of keyword arguments.

Create a new namespace called fun-functions. Define the following functions and call them with some test input:

Write a function that takes as input a state and corgi-count, and returns a boolean indicating the person’s eligibility for the “corgi cover” policy.

See if =.

A focus group of corgi owners has revealed that “corgi cover” needs to be offered at 3 different tiers: “corgi cover silver”, “corgi cover gold”, and “corgi cover platinum”. Platinum is available when covering 7 or more corgis OR covering at least 3 corgis and also having one other policy with Insuricorp. Gold is available when covering at least 3 corgis. Silver is the original “corgi cover” policy. Create a new function that takes an additional argument policy-count and returns a keyword indicating their eligibility.

See cond.

The “corgi cover” applications Insuricorp collect contain more information than necessary to determine eligibility. Create a new function that takes as input a single map data structure as input instead of multiple inputs. It should pick out the values that it needs from the input map. Create some test data and feed it to your function. The data should look something like:

Insuricorp just merged with Megacorp. Platinum level corgi cover is now offered to people with an existing Megacorp policy as well. Because the company is still restructuring, the policy-count input still only contains Insuricorp data. But a new input has been made available to you which is a map of people to policies.

Create a new function that takes as inputs two maps: the application, and the existing policies. It should apply the same logic, but make use of the Megacorp data.

You can define a test in any file, but it is common to put all test code in a separate “test” directory, and to create namespaces that mirror the “src” directory but have -test appended. So if we have a source file src/my_namespace.clj then we create a test file as test/my_namespace_test.clj.

Test namespaces are normal Clojure namespaces. Test related functions come from the clojure.test namespace, so it is common to refer all symbols from clojure.test for convenience:

A test is just a function that takes no arguments and will be called by the Clojure test runner.

You can run the tests manually from the REPL:

To run all tests in a project from the command line:

Lein-test-refresh is a Leiningen plugin that reloads code and re-runs tests when you save a file. https://github.com/jakemcc/lein-test-refresh.

Add lein-test-refresh to your ~/.lein/profiles.clj. It should look similar to below.

Alternatively you may add it to your project.clj.

Now you can watch for changes from the command line:

If you change my-test now to print a new message, the tests are re-run as soon as you save the file…​ giving immediate feedback on your change.

Seeing as saving the file executes code, you can use lein-test-refresh like a REPL.

Let’s begin with a false assertion:

And then convert it to a true assertion:

We have written a test that makes an assertion about the function inc. Most tests check for equality with the expected value first, and the actual value second. The expected value is a literal expression and the actual is a call to the function under test. However you are not limited to following this for every test case. You can use any truthy assertion. Here is an example that does not do equality checking:

If your assertion expression is not self explanatory, supply an optional string argument which describes the assertion:

And to group assertions into logical blocks, use the testing form:

It is also possible to more concisely express multiple assertions using the are form:

Occasionally we need to assert that an exception is thrown:

Test fixtures are for setting up and tearing down resources required by your tests. We can specify :once fixtures that execute one time for all tests in the namespace, or :each fixtures that run around each test in the namespace.

A fixture is simply a function that takes a test and executes it. Recall that tests are functions.

Now the test runner will print out “before”, execute the tests in the namespace, and then print out “after”.

Here we prevent printing within our function from appearing in the console. Usually we want our tests to make assertions, but not produce output. Otherwise the test report can be cluttered.

Another common use case is when doing database tests, we can wrap the test execution inside a transaction and rollback after the test completes. This avoids cleaning up data after the tests run, as no data was created.

Often when we are writing tests we want to isolate particular behaviors. Some parts of a function might not be appropriate to occur during the test. We can conveniently replace the definition of any var during a test using with-redefs:

At first glance this is very similar to let, but notice that a let would not work in this example. We changed the behavior of the str function whose definition is outside the scope of the test. We replaced it with an anonymous function that always returns “Goodbye world” regardless of its inputs. Note that we could have used (constantly "Goodbye world") instead, which produces an anonymous function just like the one we defined.

While working on a function, sometimes it is useful to print out an intermediary value. One way to accomplish this is using doto. Say that we were working on a complicated nested function:

And we wanted to see what (+ a (* a b)) was evaluating to in the context of the function call. We can temporarily wrap the expression in (doto …​ (prn)).

The difference from wrapping with just prn is that prn always returns nil, while doto will cause the prn side-effect to occur, but will return the original argument. This is also very useful when interacting with Java, because you can construct an object, call various methods on it, and return the object constructed.

To demonstrate these techniques in action let’s walk through the first 2 parts of Challenge 1 from the manual.

See Challenge 1: Corgi Cover eligibility

lein new corgi-cover creates a new project. When we run the tests, one failure is shown. In order to use test-refresh we first need to install the plugin. Global plugins are defined in ~/.lein/profiles.clj. I like to configure "changes only" because on large projects I will get more focused output. lein test-refresh listens for source code changes.

I like to dock test refresh on the right hand side and open my editor on the left. I’ll open the project in Idea. I can see the code side by side with the test output. When I make changes to the code and save, I see the results immediately. Changes in non-test namespaces also get reloaded. I don’t have a REPL running, but I can still evaluate code immediately just by saving the file. Test refresh loads source files as they change rather than relying on editor integration.

Let’s set up a function to check eligibility for a Corgi Cover insurance policy. The rule limits the set of states available, and requires at least one corgi. The states are Illanois, Washington, New York and Colorado. Pressing control-q shows docinfo on a function. Control-p shows the parameter summary for a function.

Printing the result of an expression in global scope is a handy way to experiment. This is quite useful because it replaces much of what I would otherwise do by sending forms to a REPL. I use prn because it is easy to search for and remove later. Add something attention grabbing to highlight the output in test refresh. Let’s play around with some input values and see that we get the expected output. Once the function is doing what we want, we move this example into a test. Let’s make sure that the test is effective by breaking it temporarily. Verifying that the test reports an error gives us confidence that the test is working correctly.

Part 2 is about introducing platinum, gold, and silver tiers. The hint is to use cond to specify the conditions and return the appropriate tier. Platinum requires a corgi count of 7 or more, or you have 3 corgies and an existing policy. Note that the >= operator is used in prefix notation, which may seem confusing. My rule of thumb is to look at the > arrow direction as indicating "getting smaller". The arrow is pointing to the right, so values from left to right must get smaller.

Let’s introduce an error to the code. Test refresh detects the problem and will not run the tests until we fix it. Now let’s see if we can get :platinum back from our function. If ineligible inputs are given we should return nil. We can use a shared definition of the eligible states, or call the eligible? function from part 1.

The REPL is fantastic at answering questions, and we can do the same thing using test refresh. To confirm how when behaves we can experiment. We can now confidently wrap our cond in when eligible?. Let’s add the logic for gold. Silver applies to any other eligible application.

When debugging we might be interested in what the eligibility check is returning. We can wrap that form in a doto prn to spy on the intermediary result. Prepending a label helps me identify the output in the test refresh window. In some ways this is similar to creating a breakpoint. This approach is convenient because we don’t need to restart anything manually. The working example is reloading as we make adjustments.

We can capture more requirements in separate assertions.

You can alternatively use a more REPL based approach. Let’s start up a REPL in Idea. There is an action to run the tests in the current namespace. It highlights failures in the editor. We need to switch between the test and implementation namespace quite often. There is an action to assist with this called goto test.

When working from a REPL it is convenient to create a commented form. The commented form can be sent to the REPL for evaluation. But if we forget to re-evaluate the function that the comment calls in the REPL, then any changes will not be active in the environment yet. I find it a little distracting to think about what needs to be evaluated. Delegating this responsibility to test refresh frees up my attention to focus on the code.

Another special form is if which chooses between two options.

Only one branch is evaluated, whereas a function call evaluates all arguments.

Often we want to execute some code only when a condition is met:

When the test fails, nothing is evaluated, when it passes, everything in the body is evaluated.

Cond allows for multiple branches.

Note that :else is not a special keyword, it just happens to be a truthy value.

Functions that call themselves are called recursive. Here is an example of recursion:

In Clojure there is a special way to do recursion which avoids consuming the stack:

Recur can only occur at the last position of a function (where scope can be discarded).

Loop establishes bindings, and allows you to recur back to the start of the loop with new values.

You can work with exceptions using try catch finally and throw.

Anything following a semicolon is a comment

Less common is the comment form:

And a special form for complete removal of any form it is prefixed to

Which is handy for temporarily removing a form when modifying code. You can use hash-underscore multiple times to comment out multiple forms.

I call this the bug eyes operator, because it looks like a bug emoji.

Commas are optional and treated as whitespace.

You have probably noticed that Clojure functions always return a value. Moreover they usually return a useful result, not just a nil. There is a distinction to be made between functions which produce useful result values from functions which cause side-effects.

Functions that produces side effects are often called in a way that discards their result. For example calling (println "hi") is done not because we want a result. println returns nil, which is useless. What we want is to print to System out the string "hi", which occurs as a side-effect of us calling the function. Contrast that with calling (str "hi" "there"), which returns a new string "hithere"; no side-effects occur.

A function with no side-effects is a pure function. Calling pure functions with a given input always results with the same corresponding output. Note that rand is not a pure function even though it returns a useful result, because it produces a different output every time.

Pure functions are desirable because they are:

The Clojure api provides many pure functions. For example conj does not add something to a vector, it returns a completely new vector!

In this example we can see that v remained unchanged. Clojure implements data structures that enable this to happen efficiently. Using a regular Java vector would require duplicating the vector, but Clojure makes use of a technique called shared structure to provide immutable data structures that don’t require the entire object to be duplicated.

Clojure does allow side-effects, indeed they are very useful. It is good style to keep side-effects co-located instead of having them occur throughout various parts of the code. We will see some good examples of this philosophy in action later in the course when we get to atoms. We can use pure function to calculate the next value to be assigned to an atom given the current value. The logic is separate from the side effect.

If you have 4 numbers and want the max, you can call

But what if you have a sequence of many numbers? What if you don’t know how many numbers there will be? Fortunately there is a way to convert a sequence of arguments into a function call:

This is especially useful when calling variadic functions like max. Note that we could have alternatively reduced over the sequence, but apply is much more concise and clear about the intent.

In Clojure we often pass functions as values, so there is a convenient way to create a function that consumes some arguments that can be used with additional arguments later:

Creates a function that adds 1 to any number of arguments supplied. It returns a function that is equivalent to:

So let’s see how we might make use of that:

In the previous example, we could have instead written:

To really embrace Clojure is to think in terms of sequences and data structures.

The most basic way to construct a sequence is like so:

But Clojure provides several easier ways to create a sequence:

Be careful though, Clojure can produce infinite sequences (don’t do this in a REPL):

This would attempt to keep producing numbers forever. (Press control-c to cancel the REPL if you did try this). There is a way to limit the amount of values to take:

Clojure has an excellent sequence abstraction that fits naturally into the language. From a vector [1 2 3 4] we can find the odd numbers by calling the filter function:

Here we called the filter function with two arguments: the odd? function and a vector of integers. filter is a higher order function, since it takes an input function to use in its computation. The result is a sequence of odd values. Functions like filter that operate on sequences call seq on their arguments to convert collections to sequences. The underlying mechanism is the ISeq interface, which allows many collection data structures to provide access to their elements.

map is a function that applies another function for every element in a sequence:

The result is a sequence of the increment of each number in [1 2 3 4].

Sequences can be used as input arguments to other functions as shown here:

Here we filtered by odd? the values from (2 3 4 5), which was the result of calling map.

To aggregate across a sequence, use reduce:

For each element in the sequence, reduce computes (* aggregate element) and passes the result of that as the aggregate for the next calculation. The first element 1 is used as the initial value of aggregate. The final result is 1 * 2 * 3 * 4.

Clojure provides a built-in function for grouped aggregates:

3 letter words are "the" and "fox", whereas 5 letter words are "quick" and "brown".

filter is like a Java loop:

map is like a Java loop:

reduce is like a Java loop:

Sequence abstractions are like names for loops that you can add to your vocabulary to talk about and recognize different kinds of loops. Learning the names of the abstractions and patterns that replace loops is an effort, but it adds powerful words to a programmer’s vocabulary. A large vocabulary facilitates reasoning more succinctly, communicating more effectively, and writing less code that does more.

Clojure provides a special form #() to create an anonymous function:

The % symbol is an implied input argument. This function takes one argument and returns true if the input argument is less than 3, otherwise it is false. Anonymous functions are handy for adding small snippets of logic:

This keeps only numbers less than 3. Now let’s create a sequence of odd/even labels for each number in the vector:

Sequence abstractions are more concise and descriptive than loops, especially when filtering multiple conditions, or performing multiple operations.

Clojure also has useful functions for constructing sequences:

One situation that appears difficult to use a sequence abstraction in is when we have a vector of numbers and wish to perform a sequence operation that relies upon the previous value visited. For example, think about finding the sum of each pair in [1 2 3 4 5]. Using an imperative style loop we can peek into the vector at the previous value:

Can we represent this as a sequence? Yes! Imagine two identical sequences offset slightly:

The overlapping values are the pairs we want.

map can take multiple sequences from which to pull arguments for the input function:

Here 1 adds to 2 to make 3, and 3 adds to 4 to make 7.

rest is a function which returns the input sequence without its first element:

Putting them together:

We called map on the addition function over both input sequences:

The input sequences were of different lengths, so map stopped when the smallest sequence was exhausted. The result was a new sequence of the pairwise sums:

Why are sequence abstractions better than loops? When reading a loop you must comprehend the entire block of code to know what it does. As the loop body grows and changes you must mentally keep track of more complexity. Mistakes like “off by one” are hard to spot, and can creep in as the code changes. Testing requires the invasion of the loop with breakpoints. You may find yourself duplicating a loop to customize some similar operation. The loop abstraction is very easy to understand and use, but it does not provide leverage.

Imagine discovering a new requirement where you need to multiply all of those numbers together. The change is invasive to the imperative loop:

The change occurs inside the loop with the addition and multiplication intertwined.

Contrast this with modifying the Clojure sequence. We compose a reduce with the original map expression:

This is dense, but descriptive code…​ if you know the vocabulary.

With a sequence you can write unit tests for the component sequences and operations, reuse the same sequence without writing new code, and reason about the transformations as composable parts.

Look out for opportunities to name your steps by identifying long expressions and creating a named function out of them.

Clojure exposes a sequence interface over data collections to a rich set of functions that compose well. Three important functional sequence concepts are: filter, which retains each item in a sequence where some function evaluates to be truthy; map, which selects new values by calling a function over input sequence(s) to create a new sequence; and reduce, which aggregates a sequence and returns a single value.

I invite you to take the “no loops” challenge. The next time you spot a loop stop and think about what sequence operation the loop represents. Think about how to rewrite the loop as sequence operations instead. It will take time and mental effort, but you will be rewarded with a deeper understanding of the problem being solved. Whenever you see a loop, think about how it could be expressed as a sequence. Sequences are loop abstractions that allow you to ignore the implementation details.

By now, you should be feeling the combinatorial power functions offer. Simple functions compose sequence operations together to build transforms. Clojure has almost one hundred functions related to sequences, so you should also be feeling wary of such dense code. If we keep adding layers of function calls, the code becomes cryptic:

With three layers of function calls, things are getting hard to keep in our head all at once. This expression may be easier to mentally process by starting from the innermost map, working out to filter, and then out to reduce last. But that is the opposite of our reading direction and locating the true starting point is difficult.

The presentation of sequence operations is clearer if you name intermediary results:

Or use a thread last:

Threading is good for unwrapping deeply nested function calls, or avoiding naming intermediary steps that don’t have a natural name.

Thread first is similar, but passes the value in the first position

Note that for empty expressions, the parenthesis are optional.

Maps sets vectors and keywords are functions. They delegate to get. While it is possible to use get to access collections, calling the collection directly is more common.

This is useful because you don’t need to create a function to call get.

Can instead be written as:

Where we are looking up the value associated with :a for each element in a vector of maps.

Sets implement get:

And so do vectors:

Create a function that opens a file called corgi-cover-applications.csv and converts every row into a data structure and prints it. Next use that data structure as an input to your validation function and print the result. See slurp line-seq clojure.string/split.

The downstream Insuricorp systems will only be operating on corgi cover applications that pass your eligibility check. But the invalid corgi cover applications need to be sent back to the branches so that they can follow up with the customers on why they are not eligible. Create a new function that opens two output files and writes to them based upon your eligibility check. The files should be called eligible-corgi-cover-applications.csv and ineligible-corgi-cover-applications.csv.

A request has come in from several Insuricorp branches that if a person is ineligible for corgi cover, a short reason be supplied. That way the sales reps don’t have to spend time figuring out what they need to tell the customer. Create a new validation function that instead of returning a boolean, returns nil if no problems are found, or returns a string with the reason if a problem is found. Create a new processing function that splits the applications into two files based on the new validator.

As part of the Megacorp merger, the downstream systems are converting to JSON format. Create a new function that writes JSON data to an eligible-corgi-cover-applications.json file.

Constructing a Java object is done by appending a period to the class identifier:

Which is equivalent to the less used variant:

Calling a method on a Java object done by prepending a leading period:

Which is equivalent to the less used variant:

Java static method calls are accessed by slash:

Inner classes can be accessed using the dollar symbol:

reify creates an object that conforms to an interface:

Notice that we didn’t define a class? We directly created an object that conforms to the FileFilter interface. reify is a convenient way to provide a concrete implementation of an interface.

gen-class creates a class. In practice the need to create a class from within Clojure is rare, so we won’t be covering the syntax. (see https://kotka.de/blog/2010/02/gen-class_how_it_works_and_how_to_use_it.html if you want to explore this further)

proxy can be used to extend a concrete superclass. Again the need for this is rare. (see https://kotka.de/blog/2010/03/proxy_gen-class_little_brother.html if you want to explore this further)

You can define Java classes in Java in a separate directory and add

To your project.clj file (See https://github.com/technomancy/leiningen/blob/master/doc/MIXED_PROJECTS.md for more other options.)

Vars are automatically derefed when evaluated, so it can seem like they are just a variable. But you can “see” the var itself using the var function or #' shorthand.

The most common reason you would want to do that is to examine the metadata of a var:

Metadata may be provided using ^{}

You can attach whatever metadata you wish. These are the keys the compiler looks for:

By default Vars are static. But Vars can be marked as dynamic to allow per-thread bindings. Within each thread they obey a stack discipline:

Bindings created with binding cannot be seen by any other thread. Likewise, bindings created with binding can be assigned to, which provides a means for a nested context to communicate with code before it on the call stack. This capability is opt-in only by setting a metadata tag: dynamic to true as in the code block above.

Functions defined with defn are stored in Vars, allowing for the re-definition of functions in a running program. This also enables many of the possibilities of aspect- or context-oriented programming. For instance, you could wrap a function with logging behavior only in certain call contexts or threads.

Atoms provide a way to manage shared, synchronous, independent state. They are a reference type like refs and vars. You create an atom with atom, and can access its state with deref/@. Like refs and agents, atoms support validators. To change the value of an atom, you can use swap!. A lower-level compare-and-set! is also provided. Changes to atoms are always free of race conditions.

As with all reference types, the intended use of atom is to hold one of Clojure’s immutable data structures. And, similar to ref’s alter and agent’s send, you change the value by applying a function to the old value. This is done in an atomic manner by swap! Internally, swap! reads the current value, applies the function to it, and attempts to compare-and-set! it in. Since another thread may have changed the value in the intervening time, it may have to retry, and does so in a spin loop. The net effect is that the value will always be the result of the application of the supplied function to a current value, atomically. However, because the function might be called multiple times, it must be free of side effects.

Atoms are an efficient way to represent some state that will never need to be coordinated with any other, and for which you wish to make synchronous changes (unlike agents, which are similarly independent but asynchronous).

While Vars ensure safe use of mutable storage locations via thread isolation, transactional references (Refs) ensure safe shared use of mutable storage locations via a software transactional memory (STM) system. Refs are bound to a single storage location for their lifetime, and only allow mutation of that location to occur within a transaction. In practise Refs are rarely used.

Like Refs, Agents provide shared access to mutable state. Where Refs support coordinated, synchronous change of multiple locations, Agents provide independent, asynchronous change of individual locations. Agents are bound to a single storage location for their lifetime, and only allow mutation of that location (to a new state) to occur as a result of an action. Actions are functions (with, optionally, additional arguments) that are asynchronously applied to an Agent’s state and whose return value becomes the Agent’s new state. Because actions are functions they can also be multimethods and therefore actions are potentially polymorphic. Also, because the set of functions is open, the set of actions supported by an Agent is also open, a sharp contrast to pattern matching message handling loops provided by some other languages.

Clojure’s Agents are reactive, not autonomous - there is no imperative message loop and no blocking receive. The state of an Agent should be itself immutable (preferably an instance of one of Clojure’s persistent collections), and the state of an Agent is always immediately available for reading by any thread (using the deref function or reader macro @) without any messages, i.e. observation does not require cooperation or coordination.

Agent action dispatches take the form (send agent fn args*). send (and send-off) always returns immediately. At some point later, in another thread, the following will happen:

The actions of all Agents get interleaved amongst threads in a thread pool. At any point in time, at most one action for each Agent is being executed. Actions dispatched to an agent from another single agent or thread will occur in the order they were sent, potentially interleaved with actions dispatched to the same agent from other sources. send should be used for actions that are CPU limited, while send-off is appropriate for actions that may block on IO.

Agents are integrated with the STM - any dispatches made in a transaction are held until it commits, and are discarded if it is retried or aborted. No user-code locking is involved.

Note that use of Agents starts a pool of non-daemon background threads that will prevent shutdown of the JVM. Use shutdown-agents to terminate these threads and allow shutdown.

Every Insuricorp “Corgi cover” policy application needs to be cross referenced with Megacorp to see if the customer has a Megacorp policy already via a remote web service. The web service is not available for you to test against yet. Set up a function called fetch-megacorp-policies to do the web request but leave the implementation empty. Create a test that changes the behavior of fetch-megacorp-policies to behave as though it were a web request; make it pause for 100ms before returning the policies that the person has. Set up a test that exercises the eligibility checks using the mocked version of a web request.

In Java you might write something like this:

Implement a similar solution in Clojure.

The web service you are using can handle multiple requests faster than a series of requests. It operates fastest with up to 20 connections. Modify your code such that multiple requests are made simultaneously. Compare the timing results to confirm the operations are happening in parallel.

Modify your mock of fetch-megacorp-policies such that it throws an exception randomly about 10% of the time. Make sure your tests report a failure. Now update your logic to handle the errors and retry up to 10 times. The tests should pass. Then create another test where the exception is thrown 100% of the time, and the max tries occurs.

Polymorphic dispatch. First we define the name of the multimethod, and the dispatch function:

In this case the dispatch function returns a vector pair of the species of input x and the species of input y. Now we can provide methods implementing functions to execute for a given dispatch value:

These are somewhere between a case statement and a function definition. They give the conditions under which to be called, and a function definition. Given a dispatch result of [:bunny :lion], the first method will be called on the x and y inputs, and the method here does nothing but return a value :run-away. Let’s set up some test inputs:

Now we can call encounter on the data to see what it does…​

Because keywords are functions, it’s quite common to use a keyword as a dispatch function.

A protocol is a named set of named methods and their signatures, defined using defprotocol:

No implementations are provided. Docs can be specified for the protocol and the functions. The above yields a set of polymorphic functions and a protocol object. All are namespace-qualified by the namespace enclosing the definition.

The resulting functions dispatch on the type of their first argument, and thus must have at least one argument. defprotocol is dynamic, and does not require AOT compilation. defprotocol will automatically generate a corresponding interface, with the same name as the protocol, e.g. given a protocol my.ns/Protocol, an interface my.ns.Protocol. The interface will have methods corresponding to the protocol functions, and the protocol will automatically work with instances of the interface.

Note that you do not need to use this interface with deftype, defrecord, or reify, as they support protocols directly:

A Java client looking to participate in the protocol can do so most efficiently by implementing the protocol-generated interface. External implementations of the protocol (which are needed when you want a class or type not in your control to participate in the protocol) can be provided using the extend construct:

extend takes a type/class (or interface, see below), a one or more protocol + function map (evaluated) pairs. Will extend the polymorphism of the protocol’s methods to call the supplied functions when an AType is provided as the first argument. Function maps are maps of the keywordized method names to ordinary fns. This facilitates easy reuse of existing fns and maps, for code reuse/mixins without derivation or composition.

You can implement a protocol on an interface. This is primarily to facilitate interop with the host (e.g. Java) but opens the door to incidental multiple inheritance of implementation since a class can inherit from more than one interface, both of which implement the protocol. If one interface is derived from the other, the more derived is used, else which one is used is unspecified.

The implementing fn can presume first argument is instanceof AType. You can implement a protocol on nil. To define a default implementation of protocol (for other than nil) just use Object. Protocols are fully reified and support reflective capabilities via extends?, extenders, and satisfies?. Note the convenience macros extend-type, and extend-protocol.

If you are providing external definitions inline, these will be more convenient than using extend directly

Expands into:

deftype, defrecord, and reify provide the mechanism for defining implementations of abstractions, and instances of those implementations. Resist the urge to use them to define 'structured data' as you would define classes or structures in other languages. It is preferred to use the built-in datatypes (vectors, maps, sets) to represent structured data.

Database persistence is important for many applications. We can use clojure.java.jdbc to interact with a database.

To start, create a new project

and add dependencies to your project.clj file:

Note that we need the driver we plan to use to connect to a database. In this case we are using an in memory HSQL database.

In the Clojure project we require jdbc, and set up a db connection url.

Now we are all set to start doing queries.

First we will create a table called messages inside the database with a text field named message, and then insert some rows.

And we can query the data back:

To selectively delete some data:

And now there is only one row remaining.

Let’s add some more data…​

And now we create a function to do a parameterized query.

It is important to use parameterized queries instead of string concatenation in this example because it protects us from SQL injection. Parameters are not part of the query, so they cannot perform SQL from malicious input.

If you want to redo any steps, remember that you can always drop the table and start again.

HoneySQL https://github.com/jkk/honeysql can be used to build SQL statements from data structures. This is useful when you have to programmatically combine clauses to produce a final SQL statement. For example if the user can check a checkbox to enable an additional clause in a search. In such cases it is more convenient to use Clojure’s capabilities for manipulating data structures. However if you do not need to do such manipulation, I recommend using plain old SQL queries in their original text form, as you can run them interactively from an SQL prompt much easier that way.

Using the database of your choice, set up an initial database for the Corgi Cover project. In the code, connect to the database and create the initial table required. You can use whatever schema you like, but the first requirement is to store the applications with exactly the same data as was retrieved from the file format in Challenge 2.

Modify the code to store the applications as they are processed, and the result of the eligibility check.

Ensure that all records processed from the files meets your expectations for required fields. Write a spec that explicitly defines what should be in the applications. Validate the spec on the incoming records.

Insuricorp is about to launch a new policy called “Poodle Protection”. Soon they will be processing applications with completely new rules. Set up a multimethod to handle “Poodle Protection” applications differently from “Corgi Cover” applications. For now the only difference with the rules from “Corgi Cover” is that “Poodle Protection” is available in different states: California (CA), Florida (FL), Wyoming (WY), and Hawaii (HI).

The spec library specifies the structure of data, validates or destructures it, and can generate data based on the spec. Spec was introduced into Clojure 1.9.0, so update your project.clj to the right version:

To start working with spec, require the clojure.spec.alpha namespace at the REPL:

Any function that takes a single argument and returns a truthy value is a valid predicate spec.

Sets are functions, so can be used as predicates that match one or more literal values:

Specs are registered using s/def.

A registered spec identifier can be used in place of a spec definition.

The simplest way to compose specs is with and and or. Let’s create a spec that combines several predicates into a composite spec with s/and:

We can also use s/or to specify two alternatives:

This or spec is the first case we’ve seen that involves a choice during validity checking. Each choice is annotated with a tag (here, between :name and :id) and those tags give the branches names that can be used to understand or enrich the data returned from conform and other spec functions.

Many predicates that check an instance’s type do not allow nil as a valid value (string?, number?, keyword?, etc). To include nil as a valid value, use the provided function nilable to make a spec:

Explain can be used to report why a value does not conform to a spec.

In addition to explain, you can use explain-str to receive the error messages as a string or explain-data to receive the errors as data.

Clojure programs rely heavily on passing around maps of data. Entity maps in spec are defined with keys:

Validation checks that the required attributes are included, and that every registered key has a conforming value.

Much existing Clojure code does not use maps with namespaced keys and so keys can also specify :req-un and :opt-un for required and optional unqualified keys. These variants specify namespaced keys used to find their specification, but the map only checks for the unqualified version of the keys.

In addition to the support for information maps via keys, spec also provides map-of for maps with homogenous key and value predicates.

Spec has explicit support for pre and post conditions using fdef.

The :ret spec uses fspec to declare that the returning function takes and returns a number. Even more interesting, the :fn spec can state a general property that relates the :args (where we know x) and the result we get from invoking the function returned from adder, namely that adding 0 to it should return x.

Here’s a bigger set of specs to model a game of cards:

Bad data produces errors

If we have a function deal that doles out some cards to the players we can spec that function to verify the arg and return value are both suitable data values. We can also specify a :fn spec to verify that the count of cards in the game before the deal equals the count of cards after the deal.

A key design constraint of spec is that all specs are also designed to act as generators of sample data that conforms to the spec (a critical requirement for property-based testing).

Spec generators rely on the Clojure property testing library test.check. However, this dependency is dynamically loaded and you can use the parts of spec other than gen, exercise, and testing without declaring test.check as a runtime dependency. When we wish to use these parts of spec (typically during testing), we need to declare a dev dependency on test.check in our project.clj:

The dev profile dependencies are included during testing but not published as a dependency or included in uber jars.

We require clojure.spec.gen.alpha in the ns form:

The gen function can be used to obtain the generator for any spec.

Once you have obtained a generator with gen, there are several ways to use it. You can generate a single sample value with generate or a series of samples with sample. Let’s see some basic examples:

What about generating a random player in our card game?

We can even generate an entire game:

It’s useful to spec (and generate) values in a range. For example, in the case of a range of integer values, use int-in to spec a range:

Spec also includes inst-in for a range of Dates, and double-in for double ranges.

To learn more about generators, read the test.check tutorial https://clojure.github.io/test.check/intro.html.

Spec provides a set of development and testing functionality in the clojure.spec.test.alpha namespace, which we can include with:

Instrumentation validates that the :args spec is being invoked on instrumented functions and thus provides validation for external uses of a function.

Instrument takes a fully-qualified symbol so we use ` here to resolve it in the context of the current namespace. If the function is invoked with args that do not conform with the :args spec you will see an error like this:

Instrumentation can be turned off using the complementary function unstrument. Instrumentation is useful at both development time and during testing to discover errors in calling code. It is not recommended to use instrumentation in production due to the overhead involved with checking args specs.

We mentioned earlier that clojure.spec.test.alpha provides tools for automatically testing functions. When functions have specs, we can use check, to automatically generate tests that check the function using the specs.

check will generate arguments based on the :args spec for a function, invoke the function, and check that the :ret and :fn specs were satisfied.

A keen observer will notice that ranged-rand contains a subtle bug. If the difference between start and end is very large (larger than is representable by Long/MAX_VALUE), then ranged-rand will produce an IntegerOverflowException. If you run check several times you will eventually cause this case to occur.

check also takes a number of options that can be passed to test.check to influence the test run, as well as the option to override generators for parts of the spec, by either name or path.

Imagine instead that we made an error in the ranged-rand code and swapped start and end:

This broken function will still create random integers, just not in the expected range. Our :fn spec will detect the problem when checking the var:

check has reported an error in the :fn spec. We can see the arguments passed were -3 and 0 and the return value was -5, which is out of the expected range.

To test all of the spec’ed functions in a namespace (or multiple namespaces), use enumerate-namespace to generate the set of symbols naming vars in the namespace:

And you can check all of the spec’ed functions by calling stest/check without any arguments.

While both instrument (for enabling :args checking) and check (for generating tests of a function) are useful tools, they can be combined to provide even deeper levels of test coverage.

instrument takes a number of options for changing the behavior of instrumented functions, including support for swapping in alternate (narrower) specs, stubbing functions (by using the :ret spec to generate results), or replacing functions with an alternate implementation.

Consider the case where we have a low-level function that invokes a remote service and a higher-level function that calls it.

We can spec these functions using the following specs:

And then we want to test the behavior of run-query while stubbing out invoke-service with instrument so that the remote service is not invoked:

The first call here instruments and stubs invoke-service. The second and third calls demonstrate that calls to invoke-service now return generated results (rather than hitting a service). Finally, we can use check on the higher level function to test that it behaves properly based on the generated stub results returned from invoke-service.

There is even more to spec! Once you are comfortable with the basics you can learn more at https://clojure.org/guides/spec.

Macros provide syntactic sugar. Macros first expand to produce new code that then gets compiled. The form is expanded at compile time through manipulation of the form. You can examine the expansion using macroexpand-1:

Consider two different definitions of zen:

and

Now call

The final result is the same, but notice that the input to zen1 was a list, where as the input to zen2 was the result of evaluating the list. That’s the key difference between a macro and a function.

Macros themselves are really just functions with a :macro flag set in their metadata, which causes them to be passed in the input forms unevaluated, and caused the result to be evaluated. This last part is less obvious…​ but think back to zen1…​ x was a list, we returned x, but the final result wasn’t a list…​ it was 3. The list was evaluated as a function call to +, resulting in 3.

To help write macros there is a special quoting form called syntax-quote.

Back-quote (`) Unquote (~) and Unquote-splicing (~@)

All symbols in a syntax-quote form get fully qualified.

Fully qualified symbols is desirable when creating macros, otherwise symbols may have another meaning in the context that the macro is expanded in:

Within the let block, the symbol inc has a different meaning than normal. Because m2 uses syntax quote, inc gets fully qualified to clojure.core/inc which does not collide with the let binding.

Fully qualified symbols avoids one source of collisions, but there is another:

This might seem confusing, unless you notice that:

Instead of inc operating on the input parameter, it is operating on an internal let bound value. To avoid this situation Clojure provides a let gensyms form which will produce a randomly named binding:

The let binding a# expands out to a randomly generated symbol unlikely to collide with existing symbols.

You may have noticed when we write a macro, we are really writing a function that produces code. The output is code… as data, and we manipulate code… as data. Homoiconic means that the language text has the same structure as its abstract syntax tree (AST). This allows all code in the language to be accessed and transformed as data, using the same representation. Nested code is well represented as a data structure.

When working on a non-trivial macro a good strategy is:

Stated another way; keep the macro as small as possible, and offload transformations to functions.

Create the following macros and test cases: