Clojure Conj 2012 Conference Highlights

Table of Contents

Overview

Several core themes emerged this year:

  • ClojureScript
  • Logic Programming (Core.logic)
  • Tooling and Testing

Additional summaries of the conference can be found at

Background

Sessions

LightTable, Chris Granger

http://clojure-conj.org/speakers/granger.html

  • Node Knockout project
  • build a game to evaluate language; <16m for event loop

http://nodeknockout.com/teams/kodowa http://kodowa.nko3.jitsu.com/

  • component entity system
  • entities just have ids
(component position [x y a]
           :x x
           :y y
           :a (or a 0))
  • system requires a rendering
(renderable (all-e :renderable))

This effectively turns the entire game into just data. Makes it easy to work with small compost-able functions. Not many at the conference had worked with creating games.

[:background []
:camera []
:player []]
  • seqs not fast
  • persistent collections are a GC nightmare
  • JavaScript arrays and objects are very fast
  • but can't use
  • array set and get as ? and !

Problems

  • .call failures with Closure
  • the game for KKO
  • working almost all work with ClojureScript

LightTable

http://www.chris-granger.com/2012/04/12/light-table---a-new-ide-concept/

The work with LightTable is in ClojureScript

This is possible since WebKit (Chromium) is embedded with nodejs.

https://github.com/rogerwang/node-webkit

node-webkit is one of the fundamental elements of LightTable as an implementation.

  • :behaviors
  • compostable
  • introspection by default: can see the ide, the behaviors and listeners

Behavior Object System

ClojureScript

  • errors: include the errors in the X header for the source mapping
  • compiler on nodejs rather than the jvm
  • ClojureScript analyzer should be able to look at Clojure
  • screencasts for improving developer involvement
  • tools: linode? nrepl, ritz, etc. are good points

Interesting growth in the use of nodejs and javascript with ClojureScript as a protective functionality. Games and editors are good examples are large applications but may not be the core problem area for many web applications that tend to have to look at layout rather than logic; canvas is much different.

http://blog.fogus.me/2012/04/25/the-clojurescript-compilation-pipeline/

What needs to happen to allow Node to compile

  • line numbering
  • macro support

http://elnode.org/

Typed Clojure, @ambrosebs

http://clojure-conj.org/speakers/bonnaire-sergeant.html

https://github.com/frenchy64/typed-clojure

Provide annotations (ann) on definitions (def) that indicate the propositions that are expected for the type of returns given a set of expectations.

Can you build a tool that does static type checking for Clojure.

Assumes that typing is useful but likely reasonable given Java interoperability.

Was part of the google summer of code

http://dev.clojure.org/display/community/Google+Summer+of+Code+2012

code -> (analyser / compiler: jvm) ast (java objects) -> analyze ast (maps) -> typed clojure

This is like the static analysis associat

Usage

(check-ns)

Annotation

Can annotate without changing the underlying structure.

The main tests are done with

(cf 1 (I Number Long)))
(ann-form)
(def-alias AnyInteger (U Integer ...))

Check that the first argument is the intersection (rather than U = union) of the second.

This uses ordered intersection types for

this uses a series of cases that need to be checked for correctness

Types

first

see first: has the ability to handle three cases: nil, arity type is just one (e.g., (1)), and finally the general case.

seq

Final check is nil or empty…

Variable-arity

look at the types of arity for (+) (uniform), (map, mapcat); key

Activity Streams, @tvachon

http://clojure-conj.org/speakers/vachon.html

The feed is the core experience at http://copious.com/ .

  • buy (products)
  • follow (people)
  • share (products)
  • list
  • love
  • sold
  • etc…
  • sets of stories about people and products.
  • a feed is a union of those sets.
  • could have used relational database.

Solution 1 MongoDB

This has a mongoDB back-end with the following dumps

{
actor: 'rob',
listing: 'guy-with-bear',
action: 'loved',
interested: [travis, cutter]
}

MongoDB falls over in unexpected ways. Growing interested embedded docs can lock the db.

One of the users (kaitlin) was part of the on-boarding process. Each change would generate massive documents.

  • queries required scanning the embedded docs
  • query optimizer made bad decisions

Solution 2 Redis

Digesting is hard to do: aggregating action for grouping story actors, verbs, and objects.

(defref feed-atom)
@feed-atom

Atoms allow maintaining state so the daemons can do the following:

  • thread adds feeds
  • thread expires
  • thread updates

Redis was tried but failed when too many story sets generated extensive unions.

Solution 3 Storm

https://github.com/nathanmarz/storm/wiki/Concepts https://github.com/nathanmarz/storm

spout > score > aggregate > save

spout, tuple, bolt, serialization, deserialization,

This all uses a protocol approach for Spout: nextTuple, ack, and fail.

This still used redis at the start and end of the process. Same processing bolt used for user-id+story

Only storing active users

Webfui, @lisperati

http://clojure-conj.org/speakers/barski.html

functional clientside ui framwrok with clojurScript

Author Land of Lisp

Goals

EDN / Hiccup based

Uses diffs to update a stateful representation of DOM

REPL, @jcrossley3

Create a Prismatic application that autoloads processing backends for tweets when available .

ML, @mikera

Neural Networks

Example: Scrabble

  • teach the NN what the correct score is for each character
  • (def scores)
  • letter-code is a vector of 26 characters to scores
  • (neural-network :inputs 26 :outputs 4 : hidden-sizes [6])
  • we're trying to create a function that takes a letter and outputs the score
  • define a time-chart visualization to show the number of correct scores

Example: Hand-written text

  • create an identity function that takes the same input and generates the same outputs but by using a smaller set of units (a bottleneck)
  • that is a compression layer
  • get the training examples
  • labels correspond to the training examples
  • composition: (def constructor (connect compressor decompressor)
  • use a visualization to see what the compressor looks like
  • create a coder for representing the value as a vector of binary values
  • the neural network takes the compressed 150 inputs to 10 outputs (the hand-written characters)
  • run the training then decide when the error rate is small enough to stop the training
  • show the recognition for the input training data (would show the failed cases)
  • picking the network topography isn't a hard and fast rule: create a function that creates random hidden layers and see which seems to fit

Go, @ztellman

Effectively C in Clojure…

Keynote, @richhickey

Formatting data presently:

  • XML
  • JSON
  • Protocol buffers
  • Avro
  • edn
  • Hessian
  • BERT

Context sensitivity of JSON (e.g., Date).

The distinction between the reference and the value is frequently lost in system work.

Most of the names in Clojure programs are names of functions (i.e., verbs) but systems don't like verbs (machines, storage locations).

Flow orientation rather than place orientation metaphor for factories: input raw material and output cars rather than User goes to Factory and changes state.

Queues should be fundamental to reduce coupling between system: add to queue and forget rather than have message, know end-point, know end-point status. Pub/sub is the preferred system-level approach.

Value names should never have meaning (e.g., "fred" or "fred17") since people begin to care. Example is in the context of Datomic on Riak + ZooKeeper.

Errors are things that you think should be there not the programmer convince thought of the intended functionality locally. http://www.erlang.org/download/armstrong_thesis_2003.pdf

The heterogeneous approach of simple services.

Non-program to program communication (e.g., SQL or Unix applications) requires a level of indirection (a parser).

Leiningen, @technomancy

Dependency management

lein deps :tree
lein pendantic

Package signing

# sudo emacs /etc/hosts
mv .m2 .m2.old
lein deps :verify
gpg --send-keys A2CD6D08
gpg --sign-keys $KEY_ID

ClojureScript, @ohpauleez

Managing risk is a core focus

Evaluating the fit

Example

  • Clojure + CLJS
  • Jetty
  • namespaces + functions

Risks

Debugging is still hard. Errors still require JavaScript background (and that likely won't change).

MiniKanren, @webyrd

http://clojure-conj.org/speakers/byrd-friedman.html

Petite Chez Scheme is the inferior scheme.

  • use q always
  • unification
  • fresh
  • conde
(run 1 (q) ...)
  • rember done in MiniKanren.
  • o suffix is "the top of ?"
()

Unconference

Practical monads

http://clojure.net/ http://www.intensivesystems.net/tutorials/monads_101.html

(def f (comp (partial * 2) inc))

;; First Law - Bind
(defn bind [mv f] (apply concat (map f mv)))

;; Second Law - Identity 
(= (vector 4) (bind (vector 4) vector))

;; Third Law - Associativity 
(= (bind (bind (vector 4) f) g) (bind (vector 4) (fn [x] (bind (f x) g))))
  • vector pulls out (1st)
  • bind undoes what vector does (2nd)
  • bind is associative over functions (3rd)

A monadic value is a container. For the examples noted above that container is a vector. The most obvious of these would be a hash-set.

(hash-set 9)
(defn set-bin [mv f] (apply clojure.set/union (map)))

Using this with constantly…

((constantly 3))
(defn bind [mv f] (fn [] (let [m (mv) new-mv (f m)] (new-mv))))
(defn f [x] (constantly (inc x)))
((f 4))
  • bind for constantly
  • get the value out of the new mv
  • this is a useless monad
  • these are monadic values that are functions that take no arguments
(defn result [x] (fn [state] [x state]))
((result 3) :state)
(def h (result 3))
(h #{})
  • type signature of f is that it always returns a monadic value
  • the result of f
  • bind takes values out of a monadic value and binds to a symbol that is a function
(defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state))))
(defn f [x] (fn [state] [(inc x) (conj state :inced)]))
(def h (f 8)) 
;; Gives back the function with a conjable state 
(h []) 
;; [9 [:inced]]
;; Cartesian product with bind and result 
(def v1 [:a :b :c])
(def v2 [1 2 3])
(bind v1 (fn [x] (bind v2 (fn [y] (vector [x y])))))
m-do

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CEAQFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.68.812%26rep%3Drep1%26type%3Dpdf&ei=59umUP7KF4-k8AShroFw&usg=AFQjCNFPJcG2_D6hOZQpOvHIPGkmS8lAdw&sig2=cUEjgQ3cnJJQ9mQ68GVbuw

;; TODO: Review later
(defmacro m-do [result [sym 1 mv1 sym2 mv2] expression] `(bind ~mv1 (fn [~sym1] (bind ~mv2 (fn [~sym2] (~result ~expression))))))
(m-do vecotor [x v1 y v2] [x y])

;; Reset the bind 
(defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state))))
(defn poke [k v] (fn [state] [nil (asoc state k v)]))
(defn peek [k] (fn [state] [(get state k) state]))

;; Peek into the value associated with a :a and also show the original set
(def h (peek :a))
(h {:a 6})
(def h (m-do result [_ (poke :a 3) x (peek :a)]))
List monad
State monad

Lets you pass around mutable state value without seeing the state.

  • maybe
  • imperative like DSL with control flow (recursive descent parser).

The Refined Clojurist

Testing
Finite Domains

Different in Clojure than MiniKanren and Prolog.

Examples

Annotation on data structures that allow for contstraints.

Many examples pulled from

Adding new constraints is trivial:

(defmethod prep-subst ::one-two-three...
(unifier {:foo 1 :bar 2} (partial-map {:foo 1}) )

Testing Techniques, Tools, and Directions

  • generative and simulation approaches
  • use levels of indirection of the test approach to ensure that the implementation isn't the same as the underlying test that could mask the death case
  • everything should be a sha with a pointer for reproducible tests
  • unit tests aren't used extensively with halloways form of clojure development instead focusing on exploration in the repl or checking in small complete programs that show how functionality works during the initial exploration of the intended functionality
  • driving a ui at a data level would be preferable to the drivers required for integration with something like selenium
  • tests can become more complicated that the system under development (but likely indicates an abstraction error)
  • people who generated the model can review the sim to ensure that it appears to correctly reflect the expected operation of the system
clojure.test.generative

Storm

Debuggers, @hugoduncan

http://clojure-conj.org/speakers/duncan.html https://github.com/pallet/ritz

  • uses Leiningen profile (~/.lein/profiles.clj)
  • lein repl :headless

Ritz has been refactored to include nrepl support.

{:user
 {:plugins [[lein-difftest "1.3.8"]
            [lein-marginalia "0.7.1"]
            [lein-pprint "1.1.1"]
            [lein-ritz "0.5.0"]            
            [lein-swank "1.4.4"] ]}}
lein ritz-nrepl 

Likng soruce code requries

lein pom
mvn dependency:sources

From inside of Emacs run nrepl-ritz-jack-in.

===============================================================================================
:id  | :name                       | :status   | :at-breakpoint? | :suspended? | :suspend-count
===============================================================================================
     | system                      |           |                 |             |               
     |   main                      |           |                 |             |               
1    |     main                    | :wait     | false           | false       | 0             
1665 |     JDI-VM-Control-Thread   | :running  | false           | true        | 1             
1881 |     msg-pump4597            | :wait     | false           | false       | 0             
5086 |     Poller SunPKCS11-Darwin | :sleeping | false           | false       | 0             
5082 |   Reference Handler         | :wait     | false           | false       | 0             
5083 |   Finalizer                 | :wait     | false           | false       | 0             
5084 |   Signal Dispatcher         | :running  | false           | false       | 0             
===============================================================================================

Outcomes

DONE http://emacswiki.org/emacs/volatile-highlights.el

Shown during the Data Science demo.

http://emacswiki.org/emacs/volatile-highlights.el

This also allowed for single hand navigation.

DONE solarize-dark (nolan during core.logic pres)

DONE How does one just make a tag a ClojureScript project

Document the testing model using SHA for reproduction

already implemented but note as pitch.

Push release results back to s3

Escalate that the tests for the tags are more complicated than the functionality

seen the notes on the unconference requires stepping back and building the infrastructure separately

Write ClojureScript webfui application

ClojureScript / Reflex

ML for stock market or other JavaScript features

Install nrepl for console debugging

Look at @longwaydown rsch on malicious ad-related JavaScript behavior

Script auditor

Chrome plugin for reviewing all scripts on page + Esprima and look for marks of libraries, ads, trackers, or features

Author: Jason Walsh

j@wal.sh

Last Updated: 2024-08-02 10:09:48