Clojure Conj 2012 Conference Highlights
Table of Contents
- Overview
- Background
- Sessions
- LightTable, Chris Granger
- Typed Clojure, @ambrosebs
- Activity Streams, @tvachon
- Webfui, @lisperati
- REPL, @jcrossley3
- ML, @mikera
- Go, @ztellman
- Keynote, @richhickey
- Data Science, @edmundjackson
- Leiningen, @technomancy
- ClojureScript, @ohpauleez
- MiniKanren, @webyrd
- Statistics, @lynaghk
- REPL, @kovasb
- Complexity, @mtnygard
- Unconference
- Logic, @miner
- Debuggers, @hugoduncan
- TCO, @longwaydown
- ClojureScript, @bodil
- Outcomes
- DONE Install stock Emacs
- DONE http://emacswiki.org/emacs/volatile-highlights.el
- DONE solarize-dark (nolan during core.logic pres)
- DONE How does one just make a tag a ClojureScript project
- Document the testing model using SHA for reproduction
- Push release results back to s3
- Escalate that the tests for the tags are more complicated than the functionality
- Write ClojureScript webfui application
- ClojureScript / Reflex
- ML for stock market or other JavaScript features
- Install nrepl for console debugging
- Core.logic videos
- Look at @longwaydown rsch on malicious ad-related JavaScript behavior
- Script auditor
Overview
Several core themes emerged this year:
- ClojureScript
- Logic Programming (Core.logic)
- Tooling and Testing
Additional summaries of the conference can be found at
Background
Sessions
LightTable, Chris Granger
http://clojure-conj.org/speakers/granger.html
- Node Knockout project
- build a game to evaluate language; <16m for event loop
http://nodeknockout.com/teams/kodowa http://kodowa.nko3.jitsu.com/
- component entity system
- entities just have ids
(component position [x y a] :x x :y y :a (or a 0))
- system requires a rendering
(renderable (all-e :renderable))
This effectively turns the entire game into just data. Makes it easy to work with small compost-able functions. Not many at the conference had worked with creating games.
[:background [] :camera [] :player []]
- seqs not fast
- persistent collections are a GC nightmare
- JavaScript arrays and objects are very fast
- but can't use
- array set and get as ? and !
Problems
- .call failures with Closure
- the game for KKO
- working almost all work with ClojureScript
LightTable
http://www.chris-granger.com/2012/04/12/light-table---a-new-ide-concept/
The work with LightTable is in ClojureScript
This is possible since WebKit (Chromium) is embedded with nodejs.
https://github.com/rogerwang/node-webkit
node-webkit is one of the fundamental elements of LightTable as an implementation.
- :behaviors
- compostable
- introspection by default: can see the ide, the behaviors and listeners
Behavior Object System
ClojureScript
- errors: include the errors in the X header for the source mapping
- compiler on nodejs rather than the jvm
- ClojureScript analyzer should be able to look at Clojure
- screencasts for improving developer involvement
- tools: linode? nrepl, ritz, etc. are good points
Interesting growth in the use of nodejs and javascript with ClojureScript as a protective functionality. Games and editors are good examples are large applications but may not be the core problem area for many web applications that tend to have to look at layout rather than logic; canvas is much different.
http://blog.fogus.me/2012/04/25/the-clojurescript-compilation-pipeline/
What needs to happen to allow Node to compile
- line numbering
- macro support
Typed Clojure, @ambrosebs
http://clojure-conj.org/speakers/bonnaire-sergeant.html
https://github.com/frenchy64/typed-clojure
Provide annotations (ann) on definitions (def) that indicate the propositions that are expected for the type of returns given a set of expectations.
Can you build a tool that does static type checking for Clojure.
Assumes that typing is useful but likely reasonable given Java interoperability.
Was part of the google summer of code
http://dev.clojure.org/display/community/Google+Summer+of+Code+2012
code -> (analyser / compiler: jvm) ast (java objects) -> analyze ast (maps) -> typed clojure
This is like the static analysis associat
Usage
(check-ns)
Annotation
Can annotate without changing the underlying structure.
The main tests are done with
(cf 1 (I Number Long))) (ann-form) (def-alias AnyInteger (U Integer ...))
Check that the first argument is the intersection (rather than U = union) of the second.
This uses ordered intersection types for
this uses a series of cases that need to be checked for correctness
Types
first
see first: has the ability to handle three cases: nil, arity type is just one (e.g., (1)), and finally the general case.
seq
Final check is nil or empty…
Variable-arity
look at the types of arity for (+) (uniform), (map, mapcat); key
Activity Streams, @tvachon
http://clojure-conj.org/speakers/vachon.html
The feed is the core experience at http://copious.com/ .
- buy (products)
- follow (people)
- share (products)
- list
- love
- sold
- etc…
- sets of stories about people and products.
- a feed is a union of those sets.
- could have used relational database.
Solution 1 MongoDB
This has a mongoDB back-end with the following dumps
{ actor: 'rob', listing: 'guy-with-bear', action: 'loved', interested: [travis, cutter] }
MongoDB falls over in unexpected ways. Growing interested embedded docs can lock the db.
One of the users (kaitlin) was part of the on-boarding process. Each change would generate massive documents.
- queries required scanning the embedded docs
- query optimizer made bad decisions
Solution 2 Redis
Digesting is hard to do: aggregating action for grouping story actors, verbs, and objects.
(defref feed-atom) @feed-atom
Atoms allow maintaining state so the daemons can do the following:
- thread adds feeds
- thread expires
- thread updates
Redis was tried but failed when too many story sets generated extensive unions.
Solution 3 Storm
https://github.com/nathanmarz/storm/wiki/Concepts https://github.com/nathanmarz/storm
spout > score > aggregate > save
spout, tuple, bolt, serialization, deserialization,
This all uses a protocol approach for Spout: nextTuple, ack, and fail.
This still used redis at the start and end of the process. Same processing bolt used for user-id+story
Parallelize the computation
Only storing active users
Webfui, @lisperati
http://clojure-conj.org/speakers/barski.html
functional clientside ui framwrok with clojurScript
Author Land of Lisp
Goals
EDN / Hiccup based
Uses diffs to update a stateful representation of DOM
REPL, @jcrossley3
Create a Prismatic application that autoloads processing backends for tweets when available .
ML, @mikera
Neural Networks
Only the weighted connections change not the underlying model.
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html http://www.cheshireeng.com/Neuralyst/nnbg.htm
Example: Scrabble
- teach the NN what the correct score is for each character
- (def scores)
- letter-code is a vector of 26 characters to scores
- (neural-network :inputs 26 :outputs 4 : hidden-sizes [6])
- we're trying to create a function that takes a letter and outputs the score
- define a time-chart visualization to show the number of correct scores
Example: Hand-written text
- create an identity function that takes the same input and generates the same outputs but by using a smaller set of units (a bottleneck)
- that is a compression layer
- get the training examples
- labels correspond to the training examples
- composition: (def constructor (connect compressor decompressor)
- use a visualization to see what the compressor looks like
- create a coder for representing the value as a vector of binary values
- the neural network takes the compressed 150 inputs to 10 outputs (the hand-written characters)
- run the training then decide when the error rate is small enough to stop the training
- show the recognition for the input training data (would show the failed cases)
- picking the network topography isn't a hard and fast rule: create a function that creates random hidden layers and see which seems to fit
Go, @ztellman
Effectively C in Clojure…
Keynote, @richhickey
Formatting data presently:
- XML
- JSON
- Protocol buffers
- Avro
- edn
- Hessian
- BERT
Context sensitivity of JSON (e.g., Date).
The distinction between the reference and the value is frequently lost in system work.
Most of the names in Clojure programs are names of functions (i.e., verbs) but systems don't like verbs (machines, storage locations).
Flow orientation rather than place orientation metaphor for factories: input raw material and output cars rather than User goes to Factory and changes state.
Queues should be fundamental to reduce coupling between system: add to queue and forget rather than have message, know end-point, know end-point status. Pub/sub is the preferred system-level approach.
Value names should never have meaning (e.g., "fred" or "fred17") since people begin to care. Example is in the context of Datomic on Riak + ZooKeeper.
Errors are things that you think should be there not the programmer convince thought of the intended functionality locally. http://www.erlang.org/download/armstrong_thesis_2003.pdf
The heterogeneous approach of simple services.
Non-program to program communication (e.g., SQL or Unix applications) requires a level of indirection (a parser).
Data Science, @edmundjackson
Leiningen, @technomancy
Dependency management
lein deps :tree lein pendantic
Package signing
# sudo emacs /etc/hosts mv .m2 .m2.old
lein deps :verify
gpg --send-keys A2CD6D08 gpg --sign-keys $KEY_ID
ClojureScript, @ohpauleez
Managing risk is a core focus
Evaluating the fit
http://www.thoughtworks.com/articles/technology-radar-march-2012
adopt trial hold clojure clojurescript
Example
- Clojure + CLJS
- Jetty
- namespaces + functions
Complexity
Risks
Debugging is still hard. Errors still require JavaScript background (and that likely won't change).
MiniKanren, @webyrd
http://clojure-conj.org/speakers/byrd-friedman.html
Petite Chez Scheme is the inferior scheme.
- use q always
- unification
- fresh
- conde
(run 1 (q) ...)
- rember done in MiniKanren.
- o suffix is "the top of ?"
()
Statistics, @lynaghk
REPL, @kovasb
Complexity, @mtnygard
Unconference
Practical monads
http://clojure.net/ http://www.intensivesystems.net/tutorials/monads_101.html
- Comprehending Monads (1992) http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.33.5381
- Comprehending Queries (1999) http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.88.9148
(def f (comp (partial * 2) inc)) ;; First Law - Bind (defn bind [mv f] (apply concat (map f mv))) ;; Second Law - Identity (= (vector 4) (bind (vector 4) vector)) ;; Third Law - Associativity (= (bind (bind (vector 4) f) g) (bind (vector 4) (fn [x] (bind (f x) g))))
- vector pulls out (1st)
- bind undoes what vector does (2nd)
- bind is associative over functions (3rd)
A monadic value is a container. For the examples noted above that container is a vector. The most obvious of these would be a hash-set.
(hash-set 9) (defn set-bin [mv f] (apply clojure.set/union (map)))
Using this with constantly…
((constantly 3)) (defn bind [mv f] (fn [] (let [m (mv) new-mv (f m)] (new-mv)))) (defn f [x] (constantly (inc x))) ((f 4))
- bind for constantly
- get the value out of the new mv
- this is a useless monad
- these are monadic values that are functions that take no arguments
(defn result [x] (fn [state] [x state])) ((result 3) :state) (def h (result 3)) (h #{})
- type signature of f is that it always returns a monadic value
- the result of f
- bind takes values out of a monadic value and binds to a symbol that is a function
(defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state)))) (defn f [x] (fn [state] [(inc x) (conj state :inced)])) (def h (f 8)) ;; Gives back the function with a conjable state (h []) ;; [9 [:inced]] ;; Cartesian product with bind and result (def v1 [:a :b :c]) (def v2 [1 2 3]) (bind v1 (fn [x] (bind v2 (fn [y] (vector [x y])))))
m-do
;; TODO: Review later (defmacro m-do [result [sym 1 mv1 sym2 mv2] expression] `(bind ~mv1 (fn [~sym1] (bind ~mv2 (fn [~sym2] (~result ~expression)))))) (m-do vecotor [x v1 y v2] [x y]) ;; Reset the bind (defn bind [mv f] (fn [state] (let [[v new-state] (mv state) new-mv (f v)] (new-mv new-state)))) (defn poke [k v] (fn [state] [nil (asoc state k v)])) (defn peek [k] (fn [state] [(get state k) state])) ;; Peek into the value associated with a :a and also show the original set (def h (peek :a)) (h {:a 6}) (def h (m-do result [_ (poke :a 3) x (peek :a)]))
List monad
State monad
Lets you pass around mutable state value without seeing the state.
- maybe
- imperative like DSL with control flow (recursive descent parser).
The Refined Clojurist
- A Machine-Oriented Logic Based on the Resolution Principle http://dl.acm.org/citation.cfm?id=321253
- http://en.wikipedia.org/wiki/Resolution_(logic)
- Evolution of Lisp http://dl.acm.org/citation.cfm?id=155373
- Forward Reasoning and Dependency-Directed Backtracking in a System for Computer-Aided Circuit Analysis http://dspace.mit.edu/handle/1721.1/6255
- Concepts, Techniques, and Models of Computer Programming https://mitpress.mit.edu/books/concepts-techniques-and-models-computer-programming
Testing
- http://www.gecode.org/
- SEND + MORE = MONEY is much less code
- Sodoku
Finite Domains
Different in Clojure than MiniKanren and Prolog.
Examples
Annotation on data structures that allow for contstraints.
Many examples pulled from
- Finite Domain Constraint Programming in Oz http://www.mozart-oz.org/documentation/fdt/
Adding new constraints is trivial:
(defmethod prep-subst ::one-two-three... (unifier {:foo 1 :bar 2} (partial-map {:foo 1}) )
Testing Techniques, Tools, and Directions
- generative and simulation approaches
- use levels of indirection of the test approach to ensure that the implementation isn't the same as the underlying test that could mask the death case
- everything should be a sha with a pointer for reproducible tests
- unit tests aren't used extensively with halloways form of clojure development instead focusing on exploration in the repl or checking in small complete programs that show how functionality works during the initial exploration of the intended functionality
- driving a ui at a data level would be preferable to the drivers required for integration with something like selenium
- tests can become more complicated that the system under development (but likely indicates an abstraction error)
- people who generated the model can review the sim to ensure that it appears to correctly reflect the expected operation of the system
clojure.test.generative
Storm
Logic, @miner
Prolog implementation used for http://www.tis.army.mil/PS_AALPS.htm .
http://reference.kfupm.edu.sa/content/b/i/the_birth_of_prolog__24094.pdf
- Shen, miniKanren, core.logic
- Constraint Logic Programming
- SAT solvers
- Rule-based systems
- Datalog
Debuggers, @hugoduncan
http://clojure-conj.org/speakers/duncan.html https://github.com/pallet/ritz
- uses Leiningen profile (~/.lein/profiles.clj)
- lein repl :headless
Ritz has been refactored to include nrepl support.
{:user {:plugins [[lein-difftest "1.3.8"] [lein-marginalia "0.7.1"] [lein-pprint "1.1.1"] [lein-ritz "0.5.0"] [lein-swank "1.4.4"] ]}}
lein ritz-nrepl
Likng soruce code requries
lein pom mvn dependency:sources
From inside of Emacs run nrepl-ritz-jack-in.
=============================================================================================== :id | :name | :status | :at-breakpoint? | :suspended? | :suspend-count =============================================================================================== | system | | | | | main | | | | 1 | main | :wait | false | false | 0 1665 | JDI-VM-Control-Thread | :running | false | true | 1 1881 | msg-pump4597 | :wait | false | false | 0 5086 | Poller SunPKCS11-Darwin | :sleeping | false | false | 0 5082 | Reference Handler | :wait | false | false | 0 5083 | Finalizer | :wait | false | false | 0 5084 | Signal Dispatcher | :running | false | false | 0 ===============================================================================================
TCO, @longwaydown
ClojureScript, @bodil
Outcomes
DONE Install stock Emacs
DONE http://emacswiki.org/emacs/volatile-highlights.el
Shown during the Data Science demo.
http://emacswiki.org/emacs/volatile-highlights.el
This also allowed for single hand navigation.
DONE solarize-dark (nolan during core.logic pres)
DONE How does one just make a tag a ClojureScript project
Document the testing model using SHA for reproduction
already implemented but note as pitch.
Push release results back to s3
Escalate that the tests for the tags are more complicated than the functionality
seen the notes on the unconference requires stepping back and building the infrastructure separately
Write ClojureScript webfui application
ClojureScript / Reflex
ML for stock market or other JavaScript features
Install nrepl for console debugging
Core.logic videos
Look at @longwaydown rsch on malicious ad-related JavaScript behavior
https://www.cs.indiana.edu/~minaxi/pubs/wpes12.pdf
See the associated AST tooling for http://www.cs.indiana.edu/~achauhan/Teaching/B629/2010-Fall/Assignments/manual.pdf
Script auditor
Chrome plugin for reviewing all scripts on page + Esprima and look for marks of libraries, ads, trackers, or features