Advanced Topics

Topics to be covered

  1. Use case: How to define mass data structure using spec and to get the data from that using “conform”.

    1. Need sample use case and accustomed project with that.

    2. Require more details on why we need this spec and what is the business advantage?

  2. Use case: To mock test data for resources on validating the specs as a whole data structure. (A sample project or reference available on internet is also required)

  3. How to get the project built using Jenkins or any hosting tools. We would have a demo to do that with pipeline or docker plugin.

  4. How to call web services from clojure & to hop around the different environments? How are we going to handle/pass the security configuration in between the layers?

  5. We use “:gen-class” for all the namespace definition. Is this mandatory for running the application as uber jar? What is the purpose of including this :gen-class?

  6. More examples (different usage scenarios) and elaboration required for Macros.

  7. Use case: Require to know Clojure with Hadoop interaction for data enrichment with clojure specs.

  8. Does clojure provide any utility to attach a file and send an email? Configuring logs on masking the sensitive data.

  9. How to manage real time dependency management between projects

  10. Any industry design patterns for clojure?

  11. What is Profile.clj capable of doing? Is it like getting the jar from repo, defining plugins, is that all? Or do we have more to explore. (need more clarity and reference links)

  12. What is Project.clj capable of doing to the project level and where does this scope live? Unlike the above one, require reference materials.

1) Use case: Parsing data with Spec

Insurance company "Megacorp" offers three pet insurance policies:

Can spec help us process applications for these policies?

a. Defining specs for data structures

  • See examples/parsing-with-spec

b. Validating data

(valid?) expound

c. Extracting data using conform

  • When there are multiple options to match against

Spec rationale

  • It is a guiding principle of Clojure to represent information as data (not objects)

  • Important properties of Clojure systems are represented and conveyed by the shape of the data

    • runtime types are maps, vectors, sets

    • shape is not captured or checked anywhere

  • Manual parsing and error reporting is not good enough

  • Defining specifications is optional

Spec enables automatic

  • Validation

  • Error reporting

  • Destructuring

  • Instrumentation

  • Test-data generation

  • Generative test generation

Why not spec?

  • It is new

  • Specs are code

    • however code is data and you can create specs on the fly

  • You can do without it

    • however manual approaches become difficult as complexity rises

Why spec?

  • Specifying the shape of inputs is hard

    • good error messages are hard for complex requirements

    • coming up with test data can be tedious

    • testing often involves reimplementation

  • Additional tooling

    • Expound (nice error messages)

    • Data generators

    • test.check

d. What is the business advantage of spec?

  • Implement validation quickly

  • Implement parsing quickly

  • Deliver correct, well tested solutions

  • Use the same technology in code and data interfaces

    • learn once, apply in many contexts

2) Mock test data

see examples/parsing-with-spec/test/parsing-with-spec/generated-test

(gen/sample (s/gen :mega-corp/insurance-policy))
(stest/check `my-function)

3) Continuous Integration

  • Building projects with Jenkins

    • See examples/jenkins-build-server


  • How Clojure projects are built and executed locally

  • How to set up a build server to automate test/build/deploy

Building and Executing Clojure projects

lein run
lein uberjar
java -jar myapp.jar
lein ring uberjar
lein install
lein deploy

lein run

  • Easy to execute

  • May need to pull dependencies

lein uberjar

  • Your application and all dependencies in a single JAR file

  • Easy to deploy

  • Easy to execute

  • Versioned

  • Preservable

java -jar myapp.jar

  • Executes an uberjar

  • Define a main entrypoint in project.clj

    :main myapp.core


(ns myapp.core)
(defn -main [& args] ...)
  • Alternatively, specify an entrypoint from the commandline

    java -jar myapp.jar -m myapp.core/-main

lein ring uberjar

Sets up a main entry point to start the webservice

Equivalent to

(ns myapp.core)
(defn -main [& args]
  (run-jetty handler {:port 3000}))

lein install

  • Builds an uberjar and puts it in your local Maven repository ~/.m2

  • Useful for testing library snapshots and building from source

  • Does not publish your artifact

lein deploy

Where to publish artifacts to?

Deploying a Clojure application

  • Build an uberjar (or Docker container)

  • Get the artifact to the host server

  • Run it

Implementation details are driven by architecture

  • Tomcat? Drop a WAR in a folder

  • AWS Elastic Beanstalk? Roll out a new docker container

  • AWS Lambda? Upload a new JAR

  • Kubernetes? Roll out a new docker container

  • Heroku? Deploy from git

Creating a build server overview

  • Install Jenkins

  • Install Leiningen

  • Add a build

What is Docker?

  • Virtualization

  • Dockerfile specifies a parent and setup tasks

  • Building starts with an image and runs the setup tasks, creates a container

  • Run a container and it behaves like a stand alone computer

Handy Docker commands

  • docker build -t <image-name> . to create a container from a Dockerfile

  • docker run <image-name> to run it

  • docker ps to see running containers

  • docker exec -ti <container-id> bash to get a shell in a running container

  • docker stop <container-id> to stop it

Creating a Jenkins server inside a Docker container

  • Install docker

  • Create a Dockerfile to extend the base Jenkins image

  • Create a Makefile or similar to automate tasks

    • see examples/jenkins-build-server/Makefile

    • make run to start the server

  • Open the UI: http://localhost:8080/

  • Enter password from console log

  • Install suggested plugins

  • Create admin user

What to build?

  • An uberjar

  • Docker image

Set up Job

  • Choose Freestyle project

  • Configure source code management

    • parsing-with-spec is a subdirectory of the enterprise-clojure repository

    • sparse checkout path only gets a subdirectory

  • Under "Build", add build steps, shell command

    • cd examples/parsing-with-spec && lein test

    • cd examples/parsing-with-spec && lein install

    • or docker build

    • don’t need to change directory if project is in root

  • Save

  • Build now

  • Check the console logs

    • tests passed

    • jar created


Running the built artifacts

  • java -jar myapp.jar

  • Use environment variables to behave differently

  • Can extend a prebuilt docker image (or roll your own)

Building multiple services

  • Set up jobs for each project

Set up another triggered Job to deploy to CI

  • If the build succeeds, deploy

4) Clojure services

a. Creating a webservice

lein new compojure customer-data-service

What is Ring?

  • Ring is a library that abstracts the details of HTTP into a unified API

    • modular components

  • Handlers - functions that take requests and return responses

  • Request - map of data about the request (params, body, etc)

  • Response - map containing status, headers, body

  • Middleware - the mechanism for modular components, higher order functions

Middleware pattern

Middleware are functions that return functions:

(defn wrap-user [handler]
  (fn [request]
    (if-let [user-id (-> request :session :user-id)]
      (let [user (get-user-by-id user-id)]
        (handler (assoc request :user user)))
      (handler request))))
  • Takes a handler as input

  • The function returned

    • can modify the request before passing it to the handler

    • can modify the result from the handler before returning it

    • is itself a handler


What is Compojure?

  • A routing library


    • /insurance-policy/corgi-cover is a route

    • /insurance-policy is fixed root

    • /corgi-cover is a policy ID

    • could be poodle-protection or poodle-protection-platinum

    • (GET "/insurance-policy/:id" [id] (fetch-policy id))

Useful extensions

Making HTTP requests

  • clj-http

  • See examples/communicating-services/insurance-policy-application-processor

    [clj-http "0.6.0"]
    (ns myns (:require [clj-http.client :as client]))
    (client/get (str "http://customer-data-service:3000/customer/" id))

Multiple services

See examples/communicating-services

Managing multiple services with docker-compose

See examples/communicating-services/docker-compose.yml

  • Building

  • Deploying

  • Reloading code while developing multiple services

Security and configuration

5) :gen-class

What is the purpose of :gen-class?

  • gen-class creates a Java class

  • You can specify AOT namespaces in the project.clj file

  • lein compile to build the class

  • Prefer using reify, deftype, defrecord to implement Java classes

  • Avoid gen-class and AOT

  • But why is it used so often?

Executable jars

  • An uberjar contains all your project dependencies.

  • If it contains a main class then it will also be executable:

    java -jar myuber.jar
  • A common way to make an executable uberjar is:


(ns myns.core
(defn -main [& args]
  (println "Hello World"))


:main myns.core
:aot [myns.core]

AOT is about when compilation happens

(def password (System/getenv "PASSWORD"))

Behaves differently if evaluated during AOT than during Runtime.

  • In AOT it is captured during the build prior to deployment.

  • At Runtime it is whatever is in the environment when the namespace is loaded.

When would it help to AOT?

  • Shipping a binary without the source code

  • Marginally speeding up start time

  • Generating classes loadable directly from Java for interop purposes (Hadoop)

  • Platforms such as Android do not support custom class loaders for running new bytecode at runtime.

Is :gen-class mandatory?

  • No

  • See examples/aot/too-much-aot

    • An example where aot is causing undesirable behavior

  • See examples/aot/no-aot

    • Clojure provides an entrypoint java -cp myuber.jar clojure.main -m myns.core

    • Clojure compiles all code you load on-the-fly into JVM bytecode

  • See examples/aot/little-aot

    • Avoid transitive aot by providing a bootstrap

    • Produces an executable jar

6) More on Macros

Examples (different usage scenarios) and elaboration required for Macros.

  • Many Clojure primitives such as defn are macros

  • Macros provide syntax but are not values


    • rarely

    • compile time

    • unevaluated arguments

    • inline code

  • core.async

    • macros over state machines to implement Communicating Sequential Processes (CSP)

    • provides a new syntax (go blocks)

    • available as a library

7) Data enrichment

Require to know Clojure with Hadoop interaction for data enrichment with clojure specs.

8) Sending emails

  {:host ""}
  {:from ""
   :to ""
   :subject "Hi!"
   :body [{:type "text/html"
           :content "<b>Test!</b>"}
          {:type :attachment
           :content ( "/tmp/foo.txt")}
          {:type :inline
           :content ( "/tmp/a.pdf")
           :content-type "application/pdf"}]})

You need a mail server. (AWS Simple Email Service works well

Configuring logs on masking the sensitive data.

9) Dependencies

How to manage real time dependency management between projects

10) Industry design patterns for Clojure

11) Profile.clj

What is Profile.clj capable of doing?

  • profile.clj is merged with any project.clj.

  • Having a profile.clj allows you to have common dependencies or tasks.

  • Example: lein-test-refresh plugin

Is it like getting the jar from repo, defining plugins, is that all?

  • Pretty much…​

  • Capable of anything project.clj can do, but usually just for tools.

Or do we have more to explore. (need more clarity and reference links)

Common plugins for profiles.clj

  • lein-test-refresh

  • Ancient

  • Ultra

  • humane-test-output

  • Eastwood

  • Kibit

  • Bikeshed

  • Alembic

12) Project.clj

What is Project.clj capable of doing to the project? Where does this scope live? Unlike the above one, require reference materials.