Table of Contents

1 Overview

Several core themes emerged this year:

  • ClojureScript
  • Logic Programming (Core.logic)
  • Tooling and Testing

Additional summaries of the conference can be found at

2 Background

3 Sessions

3.1 LightTable, Chris Granger

http://clojure-conj.org/speakers/granger.html

  • Node Knockout project
  • build a game to evaluate language; <16m for event loop

http://nodeknockout.com/teams/kodowa http://kodowa.nko3.jitsu.com/

  • component entity system
  • entities just have ids
(component position [x y a]
           :x x
           :y y
           :a (or a 0))
  • system requires a rendering
(renderable (all-e :renderable))

This effectively turns the entire game into just data. Makes it easy to work with small compost-able functions. Not many at the conference had worked with creating games.

[:background []
:camera []
:player []]
  • seqs not fast
  • persistent collections are a GC nightmare
  • JavaScript arrays and objects are very fast
  • but can't use
  • array set and get as ? and !

3.1.1 Problems

  • .call failures with Closure
  • the game for KKO
  • working almost all work with ClojureScript

3.1.2 LightTable

http://www.chris-granger.com/2012/04/12/light-table---a-new-ide-concept/

The work with LightTable is in ClojureScript

This is possible since WebKit (Chromium) is embedded with nodejs.

https://github.com/rogerwang/node-webkit

node-webkit is one of the fundamental elements of LightTable as an implementation.

  • :behaviors
  • compostable
  • introspection by default: can see the ide, the behaviors and listeners

3.1.3 Behavior Object System

3.1.4 ClojureScript

  • errors: include the errors in the X header for the source mapping
  • compiler on nodejs rather than the jvm
  • ClojureScript analyzer should be able to look at Clojure
  • screencasts for improving developer involvement
  • tools: linode? nrepl, ritz, etc. are good points

Interesting growth in the use of nodejs and javascript with ClojureScript as a protective functionality. Games and editors are good examples are large applications but may not be the core problem area for many web applications that tend to have to look at layout rather than logic; canvas is much different.

http://blog.fogus.me/2012/04/25/the-clojurescript-compilation-pipeline/

3.1.5 What needs to happen to allow Node to compile

  • line numbering
  • macro support

http://elnode.org/

3.2 Typed Clojure, @ambrosebs

http://clojure-conj.org/speakers/bonnaire-sergeant.html

https://github.com/frenchy64/typed-clojure

Provide annotations (ann) on definitions (def) that indicate the propositions that are expected for the type of returns given a set of expectations.

Can you build a tool that does static type checking for Clojure.

Assumes that typing is useful but likely reasonable given Java interoperability.

Was part of the google summer of code

http://dev.clojure.org/display/community/Google+Summer+of+Code+2012

code -> (analyser / compiler: jvm) ast (java objects) -> analyze ast (maps) -> typed clojure

This is like the static analysis associat

3.2.1 Usage

(check-ns)

3.2.2 Annotation

Can annotate without changing the underlying structure.

The main tests are done with

(cf 1 (I Number Long)))
(ann-form)
(def-alias AnyInteger (U Integer ...))

Check that the first argument is the intersection (rather than U = union) of the second.

This uses ordered intersection types for

this uses a series of cases that need to be checked for correctness

3.2.3 Types

3.2.3.1 first

see first: has the ability to handle three cases: nil, arity type is just one (e.g., (1)), and finally the general case.

3.2.3.2 seq

Final check is nil or empty…

3.2.4 Variable-arity

look at the types of arity for (+) (uniform), (map, mapcat); key

3.3 Activity Streams, @tvachon

http://clojure-conj.org/speakers/vachon.html

The feed is the core experience at http://copious.com/ .

  • buy (products)
  • follow (people)
  • share (products)
  • list
  • love
  • sold
  • etc…
  • sets of stories about people and products.
  • a feed is a union of those sets.
  • could have used relational database.

3.3.1 Solution 1 MongoDB

This has a mongoDB back-end with the following dumps

{
actor: 'rob',
listing: 'guy-with-bear',
action: 'loved',
interested: [travis, cutter]
}

MongoDB falls over in unexpected ways. Growing interested embedded docs can lock the db.

One of the users (kaitlin) was part of the on-boarding process. Each change would generate massive documents.

  • queries required scanning the embedded docs
  • query optimizer made bad decisions

3.3.2 Solution 2 Redis

Digesting is hard to do: aggregating action for grouping story actors, verbs, and objects.

(defref feed-atom)
@feed-atom

Atoms allow maintaining state so the daemons can do the following:

  • thread adds feeds
  • thread expires
  • thread updates

Redis was tried but failed when too many story sets generated extensive unions.

3.3.3 Solution 3 Storm

https://github.com/nathanmarz/storm/wiki/Concepts https://github.com/nathanmarz/storm

spout > score > aggregate > save

spout, tuple, bolt, serialization, deserialization,

This all uses a protocol approach for Spout: nextTuple, ack, and fail.

This still used redis at the start and end of the process. Same processing bolt used for user-id+story

3.3.3.1 Parallelize the computation
3.3.3.2 Only storing active users

3.4 Webfui, @lisperati

http://clojure-conj.org/speakers/barski.html

functional clientside ui framwrok with clojurScript

Author Land of Lisp

3.4.1 Goals

EDN / Hiccup based

3.4.2 Uses diffs to update a stateful representation of DOM

3.5 REPL, @jcrossley3

Create a Prismatic application that autoloads processing backends for tweets when available .

3.6 ML, @mikera

3.6.1 Neural Networks

3.6.2 Example: Scrabble

  • teach the NN what the correct score is for each character
  • (def scores)
  • letter-code is a vector of 26 characters to scores
  • (neural-network :inputs 26 :outputs 4 : hidden-sizes 6)
  • we're trying to create a function that takes a letter and outputs the score
  • define a time-chart visualization to show the number of correct scores

3.6.3 Example: Hand-written text

  • create an identity function that takes the same input and generates the same outputs but by using a smaller set of units (a bottleneck)
  • that is a compression layer
  • get the training examples
  • labels correspond to the training examples
  • composition: (def constructor (connect compressor decompressor)
  • use a visualization to see what the compressor looks like
  • create a coder for representing the value as a vector of binary values
  • the neural network takes the compressed 150 inputs to 10 outputs (the hand-written characters)
  • run the training then decide when the error rate is small enough to stop the training
  • show the recognition for the input training data (would show the failed cases)
  • picking the network topography isn't a hard and fast rule: create a function that creates random hidden layers and see which seems to fit

3.7 Go, @ztellman

Effectively C in Clojure…

3.8 Keynote, @richhickey

Formatting data presently:

  • XML
  • JSON
  • Protocol buffers
  • Avro
  • edn
  • Hessian
  • BERT

Context sensitivity of JSON (e.g., Date).

The distinction between the reference and the value is frequently lost in system work.

Most of the names in Clojure programs are names of functions (i.e., verbs) but systems don't like verbs (machines, storage locations).

Flow orientation rather than place orientation metaphor for factories: input raw material and output cars rather than User goes to Factory and changes state.

Queues should be fundamental to reduce coupling between system: add to queue and forget rather than have message, know end-point, know end-point status. Pub/sub is the preferred system-level approach.

Value names should never have meaning (e.g., "fred" or "fred17") since people begin to care. Example is in the context of Datomic on Riak + ZooKeeper.

Errors are things that you think should be there not the programmer convince thought of the intended functionality locally. http://www.erlang.org/download/armstrong_thesis_2003.pdf

The heterogeneous approach of simple services.

Non-program to program communication (e.g., SQL or Unix applications) requires a level of indirection (a parser).

3.10 Leiningen, @technomancy

3.10.1 Dependency management

lein deps :tree
lein pendantic

3.10.2 Package signing

# sudo emacs /etc/hosts
mv .m2 .m2.old
lein deps :verify
gpg --send-keys A2CD6D08
gpg --sign-keys $KEY_ID

3.11 ClojureScript, @ohpauleez

3.11.1 Managing risk is a core focus

3.11.2 Evaluating the fit

3.11.3 Example

  • Clojure + CLJS
  • Jetty
  • namespaces + functions

3.11.5 Risks

Debugging is still hard. Errors still require JavaScript background (and that likely won't change).

3.12 MiniKanren, @webyrd

http://clojure-conj.org/speakers/byrd-friedman.html

Petite Chez Scheme is the inferior scheme.

  • use q always
  • unification
  • fresh
  • conde
(run 1 (q) ...)
  • rember done in MiniKanren.
  • o suffix is "the top of ?"

3.16 Unconference

3.16.1 Practical monads

http://clojure.net/ http://www.intensivesystems.net/tutorials/monads_101.html

(def f (comp (partial * 2) inc))

;; First Law - Bind
(defn bind [mv f] (apply concat (map f mv)))

;; Second Law - Identity
(= (vector 4) (bind (vector 4) vector))

;; Third Law - Associativity
(= (bind (bind (vector 4) f) g) (bind (vector 4) (fn [x] (bind (f x) g))))
  • vector pulls out (1st)
  • bind undoes what vector does (2nd)
  • bind is associative over functions (3rd)

A monadic value is a container. For the examples noted above that container is a vector. The most obvious of these would be a hash-set.

(hash-set 9)
(defn set-bin [mv f] (apply clojure.set/union (map)))

Using this with constantly…

((constantly 3))
(defn bind [mv f] (fn [] (let [m (mv) new-mv (f m)] (new-mv))))
(defn f [x] (constantly (inc x)))
((f 4))
  • bind for constantly
  • get the value out of the new mv
  • this is a useless monad
  • these are monadic values that are functions that take no arguments
(defn result [x] (fn [state] [x state]))
((result 3) :state)
(def h (result 3))
(h #{})
  • type signature of f is that it always returns a monadic value
  • the result of f
  • bind takes values out of a monadic value and binds to a symbol that is a function
(defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state))))
(defn f [x] (fn [state] [(inc x) (conj state :inced)]))
(def h (f 8))
;; Gives back the function with a conjable state
(h [])
;; [9 [:inced]]
;; Cartesian product with bind and result
(def v1 [:a :b :c])
(def v2 [1 2 3])
(bind v1 (fn [x] (bind v2 (fn [y] (vector [x y])))))
3.16.1.1 m-do

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CEAQFjAB&url=http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.812&rep=rep1&type=pdf&ei=59umUP7KF4-k8AShroFw&usg=AFQjCNFPJcG2_D6hOZQpOvHIPGkmS8lAdw&sig2=cUEjgQ3cnJJQ9mQ68GVbuw

;; TODO: Review later
(defmacro m-do [result [sym 1 mv1 sym2 mv2] expression] `(bind ~mv1 (fn [~sym1] (bind ~mv2 (fn [~sym2] (~result ~expression))))))
(m-do vecotor [x v1 y v2] [x y])

;; Reset the bind
(defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state))))
(defn poke [k v] (fn [state] [nil (asoc state k v)]))
(defn peek [k] (fn [state] [(get state k) state]))

;; Peek into the value associated with a :a and also show the original set
(def h (peek :a))
(h {:a 6})
(def h (m-do result [_ (poke :a 3) x (peek :a)]))
3.16.1.2 List monad
3.16.1.3 State monad

Lets you pass around mutable state value without seeing the state.

  • maybe
  • imperative like DSL with control flow (recursive descent parser).

3.16.2 The Refined Clojurist

3.16.2.1 Testing
3.16.2.2 Finite Domains

Different in Clojure than MiniKanren and Prolog.

3.16.2.3 Examples

Annotation on data structures that allow for contstraints.

Many examples pulled from

Adding new constraints is trivial:

(defmethod prep-subst ::one-two-three...
(unifier {:foo 1 :bar 2} (partial-map {:foo 1}) )

3.16.3 Testing Techniques, Tools, and Directions

  • generative and simulation approaches
  • use levels of indirection of the test approach to ensure that the implementation isn't the same as the underlying test that could mask the death case
  • everything should be a sha with a pointer for reproducible tests
  • unit tests aren't used extensively with halloways form of clojure development instead focusing on exploration in the repl or checking in small complete programs that show how functionality works during the initial exploration of the intended functionality
  • driving a ui at a data level would be preferable to the drivers required for integration with something like selenium
  • tests can become more complicated that the system under development (but likely indicates an abstraction error)
  • people who generated the model can review the sim to ensure that it appears to correctly reflect the expected operation of the system
3.16.3.1 clojure.test.generative

3.16.4 Storm

3.18 Debuggers, @hugoduncan

http://clojure-conj.org/speakers/duncan.html https://github.com/pallet/ritz

  • uses Leiningen profile (~/.lein/profiles.clj)
  • lein repl :headless

Ritz has been refactored to include nrepl support.

{:user
 {:plugins [[lein-difftest "1.3.8"]
            [lein-marginalia "0.7.1"]
            [lein-pprint "1.1.1"]
            [lein-ritz "0.5.0"]
            [lein-swank "1.4.4"] ]}}
lein ritz-nrepl

Likng soruce code requries

lein pom
mvn dependency:sources

From inside of Emacs run nrepl-ritz-jack-in.

===============================================================================================
:id  | :name                       | :status   | :at-breakpoint? | :suspended? | :suspend-count
===============================================================================================
     | system                      |           |                 |             |
     |   main                      |           |                 |             |
1    |     main                    | :wait     | false           | false       | 0
1665 |     JDI-VM-Control-Thread   | :running  | false           | true        | 1
1881 |     msg-pump4597            | :wait     | false           | false       | 0
5086 |     Poller SunPKCS11-Darwin | :sleeping | false           | false       | 0
5082 |   Reference Handler         | :wait     | false           | false       | 0
5083 |   Finalizer                 | :wait     | false           | false       | 0
5084 |   Signal Dispatcher         | :running  | false           | false       | 0
===============================================================================================

4 Outcomes

4.2 DONE http://emacswiki.org/emacs/volatile-highlights.el

Shown during the Data Science demo.

http://emacswiki.org/emacs/volatile-highlights.el

This also allowed for single hand navigation.

4.3 DONE solarize-dark (nolan during core.logic pres)

4.4 DONE How does one just make a tag a ClojureScript project

4.5 Document the testing model using SHA for reproduction

already implemented but note as pitch.

4.6 Push release results back to s3

4.7 Escalate that the tests for the tags are more complicated than the functionality

seen the notes on the unconference requires stepping back and building the infrastructure separately

4.8 Write ClojureScript webfui application

4.9 ClojureScript / Reflex

4.10 ML for stock market or other JavaScript features

4.11 Install nrepl for console debugging

4.13 Look at @longwaydown rsch on malicious ad-related JavaScript behavior

4.14 Script auditor

Chrome plugin for reviewing all scripts on page + Esprima and look for marks of libraries, ads, trackers, or features

Author: Jason Walsh

Created: 2016-11-01 Tue 12:12

Validate