I wanted an easy way of running Clojure programs under Hadoop, and ended up writing a silly, simple little shim that would simply take a Clojure file with a mapper and reducer function. It means that there is no .JAR building and no AOT compiling — just write your mapper and reducer function, bundle up the file, and go. Note that the JAR is built for Hadoop 0.20 and later.

Getting The Jar

The quickest way of downloading the .JAR is to just download it.

Building The Jar

Build the shim:

javac -cp /usr/lib/hadoop/hadoop-core.jar:/usr/lib/hadoop/lib/*:lib/* -d classes Shim.java

Create a lib directory and add the clojure classes:

mkdir lib
cp /from/wherever/clojure-1.2.0.jar lib
cp /from/wherever/clojure-contrib-1.2.0.jar lib

Bundle it together into one .JAR file:

jar -cvf shim.jar -C classes/ . lib/*

[~/github/hadoop-shim] $ jar tf shim.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/brool/
com/brool/Shim.class
com/brool/Shim$Reduce.class
com/brool/Shim$Map.class
lib/clojure-1.2.0.jar
lib/clojure-contrib-1.2.0.jar

Using The Jar

To run the example wordcount:

hadoop jar shim.jar com.brool.Shim -files wordcount.clj input-file output-file

The output from the run will be in output-file/part-r-00000

More Details

The Clojure file that you provide should be in the user namespace, and must provide two functions named mapper and reducer.

Mapper

The mapper function takes a string representing one line in the input file and returns a list of [ key value ] pairs. A given input line can generate any number of map lines.

Reducer

The reducer is given a key and all values that were associated with that key. The reducer’s function is to consolidate all of that into one output line.

Example

Since word count is the canonical example, let’s do that. The mapper for word count is:

(defn mapper [v]
  (let [words (enumeration-seq (StringTokenizer. v))]
    (map #(do [ % "1" ] ) words)
))

As an example, for an input line of “This is a test, is it not?”, the following map will be generated:

(["This" "1"] ["is" "1"] ["a" "1"] ["test," "1"] ["is" "1"] ["it" "1"] ["not?" "1"])

Before being given the reducer, the Hadoop system will group all like keys together, resulting in:

"This" => [ "1" ]
"is"     => [ "1" "1" ]

So on and so forth. The reducer is given the key and the list of all values that had that key, and then emits the final result — for a word count, the correct code would be:

(defn reducer [k v]
  [ k (reduce + (map #(Integer/parseInt %) v)) ])

This simply adds up the counts.

Testing

In wordcount.clj there are two handy functions that allow for debugging of the mapper and reducer before submitting it to Hadoop.

Given a local file, the test-mapper will load the file and run it through the mapper; if your file is large you may just want to (take 20 (test-mapper “/my/filename”)).

The test-reducer function will load the file and run it through both the mapper and the reducer. Taking the example sentence above:

user> (test-reducer "/tmp/one-sentence")
(["This" 1] ["a" 1] ["is" 2] ["it" 1] ["not?" 1] ["test," 1])

Example #2

As another example: let’s say that you have a collection of log entries, and would like to record the first and last log entry for every user. Assume that the files are in a CSV format, with the fields being in the order of timehit, userid. Example:

2010-10-04 13:04:22,112334
2010-10-04 10:04:22,182994
2010-10-04 10:05:18,182994
2010-10-04 10:07:19,182994
2010-10-04 13:28:41,112334
2010-10-04 10:09:22,182994
2010-10-04 13:56:22,112334
2010-10-04 11:30:01,182994

The mapper for this:

(defn mapper [v]
    (let [[timehit userid] (.split v ",")]
        [ [ userid timehit ] ]
))

The reducer:

(defn reducer [k v]
(let [s (sort v)]
[k (str (first s) "," (last s))]))

The source is all on Github.

Beautiful Soup is a really nice Python library for extracting content from possibly-sloppy HTML, and I wanted some reasonably close Clojure equivalent. Unfortunately, the standard classes don’t work well with malformed HTML; as an example:

    => (require '(clojure [xml :as xml]))
    => (xml/parse "http://www.google.com")
    org.xml.sax.SAXParseException: The markup in the document preceding the root element must be well-formed. (NO_SOURCE_FILE:0)

Fortunately, there is already a TagSoup library that can parse non-perfect HTML, and it is very easy to integrate TagSoup into xml/parse. This module hardly does anything; it simply adds a few helper routines and brings the most-used calls into one amazingly bad namespace name.

Examples

Building your soup:

    (use beaujiful-soup.core)
 
    ; build soup from URL
    (def t (build-soup "http://www.google.com"))
 
    ; build soup from (deliberately malformed) string
    (def t2 (build-string-soup "<html><body><ul><li>One<li>Two</ul></body></html>"))

Extracting information is done with the xml-> call. Oftentimes the last thing you do will be a node or text or (attr :attribute) call, in order to convert the results into a more workable type:

    ; you can "walk" down the tree with successive tag names.  For
    ; example, get every list item inside the unordered list
    ; immediately inside the body.
    (xml-> t2 :body :ul :li node)
    ; => ({:tag :li, :attrs nil, :content ["One"]} {:tag :li, :attrs nil, :content ["Two"]})
 
    ; get the text for the list items
    (xml-> t2 :body :ul :li text)
    ; => ("One" "Two")
 
    ; Get textareas immediately inside the body.
    (xml-> t :body :textarea node)
    ; => ({:tag :textarea, :attrs {:id "csi", :style "display:none"}, :content nil})
 
    ; use descendants to iterate through all nodes, not just the immediate children.
    ; Get the text from all <a> tags anywhere in the body.
    (xml-> t descendants :a text)
    ; => ("Images" "Videos" "Maps" ...)
 
    ;  Get the href attribute from all tags
    (xml-> t descendants :a (attr :href))
    ; => ("http://www.google.com/imghp?hl=en&tab=wi" ... )

Use the (attr=) predicate to match an attribute value:

    ; find invisible stuff
    (xml-> t2 descendants (attr= :style "display:none") tag)
    ; => (:textarea :iframe)

Strings match the text inside nodes:

    ; find the link for the <a> that has "Videos" for content
    (xml-> t descendants :a "Videos" (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Arbitrary predicates can be used as well. They will take a loc (location), and are usually converted to a node before being used:

    ; find any :p or :div
    (defn p-or-div [loc] (contains? #{:p :div} (:tag (node loc))))
    (xml-> t descendants p-or-div tag)
    ; => (:div :div :div :div :div :div :div :div :div :div :div :p :div :div)
 
    ; find the link for <a> that has case-insensitive "Videos" for content
    (require 'clojure.string)
    (defn f [loc] 
      (let [n (node loc)]
       (and (= (:tag n) :a) (= (clojure.string/upper-case (first (:content n))) "VIDEOS"))))
    (xml-> t descendants f (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Fundamentally, the xml-> call returns a list of locations, and you can apply arbitrary transforms as necessary. For example, let’s say that you want to build a map of text => hrefs for all of the links:

    (defn loc-to-pair [loc]
        [ (attr loc :href), (text loc) ])
    (apply hash-map (xml-> t descendants :a loc-to-pair))
    ; => {"/services/" "Business Solutions",  ... }

Having a vector in the chain applies all the predicates within the vector, and filters out anything that doesn’t match. It acts a little like a lookahead in a regex. For example:

    ; Find the IDs of all divs that contain an href immediately within them
    (xml-> t descendants :div [ :a ] (attr :id))
    ; => ("fll")
 
    ; Find the IDs of all divs that contains an href anywhere within them
    (xml-> t descendants :div [ descendants :a ] (attr :id))
    ; => ("ghead" "gbar" "guser" "fll")

Source

(defproject mydirectory "1.0.0-SNAPSHOT" :description "FIXME: write" 
  :dependencies [[org.clojure/clojure "1.1.0-alpha-SNAPSHOT"] 
    [org.clojure/clojure-contrib "1.0-SNAPSHOT"]
    [mysql/mysql-connector-java "5.1.6"]
    [incanter/incanter "1.0-master-SNAPSHOT"]])

(that is, add the mysql and incanter dependencies).

Download all the dependencies with lein deps

Start up a REPL with everything in the classpath by just using lein repl

Wow, that’s kind of nice.

I really enjoy Clojure. Everything seems so well thought out and well designed; in a lot of ways it reminds me of Python, which is three ways to ironic because back in the day I started using Python because it was very Lispy. The cycle of life turns even on computer languages.

But unfortunately, I get the sense that there’s probably only room for one JVM language to make it big. As much as I like Clojure, I think that Scala will probably be the one that succeeds. It’s got the static typing, it’s got the Java-like performance, and it’s got a syntax that’s close-ish to Java, so it fills the niche that Java fills so well: a programming language that is ideal for large groups of developers.

However, it’s early enough in the development of Clojure that there’s the sense that maybe individual efforts could make a difference. (As a counterexample, Haskell is too far along — it seems like everything has been done already, or anything that needs to be done requires a deep knowledge of category theory and the latest functional design papers.)

Assuming that I would want to help out Clojure, what could be done?

Donate? Actually, I did that. I figured that I got as many hours of enjoyment out of Clojure as I did out of a typical game, so I contributed the price of a game.

Libraries? Well, I’ve written one or two. Will probably write more.

If Clojure is going to succeed, it needs a niche, an area in which it performs so well that it blows everything else away. Concurrent programming is one possibility, but there is a lot of competition there.

I have a modest proposal:

Clojure should be more like PHP.

Yes, yes, I know that programming in PHP is socially reprehensible, just one step above people that shop without their shirt while in Walmart and car salesmen.

But what if we were to give Clojure the trappings that make starting developers try it? Metaphorically, it’s like Gulliver’s Travels; it looks like a humorous, light-hearted adventure story, but underneath it’s biting political satire and deep thoughts. We lure developers in with the possibility of an easy, light-hearted way to write web sites… and then once they’re in there, once they’ve progressed from manipulating web sites and start to realize the benefits of advanced languages, they can deal with a sane language, built on top of the JVM and with the full power of a Lisp.

So call this Gulliver.

Why Did PHP Succeed?

My impression, based on absolutely no citations or hard evidence, is that PHP became popular primarily because it is:

• Easy to start trying — start with an HTML page, and just add one statement to start playing around with PHP itself
• Easy to access data — typically, with MySQL
• Easy to deploy — just upload a file and go

So, continuing the gedankenexperiment: if we were designing something that would be very approachable and very easy to use, what would it look like?

Making Compojure More Approachable

Compojure is an excellent base to start with, but it strays away from templates, and instead tries to do everything with compojure.html, a library that allows you to form everything with vectors and lists:

(html [:p "2 + 2 = " (+ 2 2)])

Handy? Yes, if you’re a developer. But if a web designer is doing your HTML, or if you’re just starting out programming and you already know HTML, though, it seems an additional barrier to entry. So what if we turn it inside out and do it like PHP does it?

<p>2 + 2 = <?= (+ 2 2) ?>

Trying something more complicated, in compojure.html this:

(html [:ul (map (fn [x] [:li x]) (range 10))])

<ul>
  <? (dotimes [x 10] ?>
  <li><?= x ?>
  <? ) ?>
</ul>

A little bit longer, but treating it as HTML makes it conceptually easier for beginning programmers to understand, especially if you want to do something like changing the id or class with every row so your CSS can make it all pretty:

<ul>
  <? (dotimes [x 10] ?>
  <li id="list-<?= x ?>" class="<?= (if (even? x) "even" "odd") ?>"><?= x ?></li>
  <? ) ?>
</ul>

Doing this is compojure.html is trickier:

(html 
  [:ul 
    (map (fn [x] [:li {:id (str "list-" x) :class (if (even? x) "even" "odd")} x]) 
    (range 10))])

So, let’s lure people in with easy to use, and they can pick up vectors and S-exprs and notational convenience later. Converting a template file into the basic code is easy:

(defn- transform-string
  "Internal.  Transform a string into the appropriate echo statement.
An empty string translates into a single space."  
  [s]
  (if (> (.length s) 0)
    (format "(gulliver/echo %s)" (pr-str s))
  " "))
 
(defn- transform-code
  "Internal.  Transform code so that it can run as a render routine."
  [s]
  (if (= (first s) \=)
    (format "(gulliver/echo %s)" (.substring s 1))
    s))
 
(defn- convert* 
  "Internal.  Given a string that represents a Clojure template,
produce a flat list of the statements necessary to produce that template."
  [s]
  (let [m (re-matcher #"(?s)(.*?)<\?(.+?)\?>|(.+?)$" s)]
    (loop [next (re-find m) buffer ["'("]]
      (cond
       (not (nil? (nth next 3))) 
           (eval (read-string (apply str (conj buffer (transform-string (nth next 3)) ")" ))))
       :otherwise (recur 
                   (re-find m) 
                   (conj buffer 
                         (transform-string (nth next 1)) 
                         (transform-code (nth next 2))))
))))

Resulting in…

==> (convert* "<ul><? (dotimes [x 10] ?><li><?= x ?></li><? ) ?></ul>")
 
((gulliver/echo "<ul>") (dotimes [x 10] (gulliver/echo "<li>") (gulliver/echo x) (gulliver/echo "</li>")) (gulliver/echo "</ul>"))

Once you have the basic code generation, it’s easy enough to hook it up to a default servlet that automatically invokes it when necessary. There’s a fair bit of handwaving going on here — regexes are questionable for this, there are some interesting issues with namespaces, and it would require some method of resolving dependencies between files, but heck, it’s only 100 lines of code; it was really quick to put together and gives us a sense of whether this can work out.

So, with the source on github, your basic server becomes…

(use 'gulliver)
(defserver web-server
  {:port 8080}
  "/*"  (servlet gulliver/template-servlet))

Easy! As an test, an example file:

<?
(import 'java.util.Date)
?>
 
<html>
  <body>
    <h1>Delivered via Gulliver</h1>
    <p>Expressions at work: <?= (+ 2 2) ?></p>
    <p>Looping:</p>
    <ul>
    <? (dotimes [x 10] ?>
      <li><?= x ?></li>
    <? ) ?>
    </ul>
 
    <p>Here's a date:  <?= (Date.) ?></p>
  </body>
</html>

… does the right thing. How about a time test?

<html>
    <body>
    <h1>Starting time test...</h1>
    <? (let [ start (System/currentTimeMillis) ] ?>
    <p>Adding 1..100000:  <?= (reduce + 0 (range 1 1000001)) ?></p>
    <p>(took <?= (- (System/currentTimeMillis) start) ?> ms)</p>
    <? ) ?> 
    </body>
</html>

… runs in about 50ms. PHP takes about 130ms on the same sequence on my laptop, so, while there are tons of optimizations we could do (like not doing a stat on the file every time the page is rendered (!)), it nonetheless performs well enough that maybe this is a viable option. Finally, as a special bonus, a stupid eval trick that evaluates Clojure code:

<html>
    <body>
        <? (when ((request :params) :code) ?>
        <pre style="border: 1px dotted">
        <?= (str (eval (read-string ((request :params) :code)))) ?>
        </pre>
        <? ) ?>
        <form method="post">
        <textarea name="code" rows=10 cols=80></textarea><br/>
        <input type="submit" value="Evaluate" />
        </form>
    </body>
</html>

Take that, non-dynamic languages!

Github copy of the source

Next part: Schema-ing Against MySQL

public void OnIdleStateChange(AccSession arg0, int arg1) {
}
 
public void OnInstanceChange(AccSession arg0, AccInstance arg1, AccInstance arg2, AccInstanceProp arg3) {
}
 
public void OnLookupUsersResult(AccSession arg0, String[] arg1, int arg2, AccResult arg3, AccUser[] arg4) {
}
 
public void OnSearchDirectoryResult(AccSession arg0, int arg1, AccResult arg2, AccDirEntry arg3) {
}
 
// ... go on like this for pages

The Java code is here, if you’re interested in the entire set of calls. Now, I didn’t care about most of those events, but I had to override them, since they didn’t have a default implementation. What made this seem painful was that I was really only interested in two of the callbacks. So I started to record an Emacs macro to convert the Java code to the equivalent Clojure proxy statement, and then I realized that I didn’t have to — I was using a Lisp.

(defmacro auto-proxy [interfaces variables & args]
  (let [defined (set (map #(str (first %)) args))
        names (fn [i] (map #(.getName %) (.getMethods i)))
        all-names (into #{} (apply concat (map names (map resolve interfaces))))
        undefined (difference all-names defined) 
        auto-gen (map (fn [x] `(~(symbol x) [& ~'args])) undefined)]
    `(proxy ~interfaces ~variables ~@args ~@auto-gen)))

Auto-proxy works just like proxy, but it makes an empty implementation for any call that wasn’t defined. So, suddenly, what would have been a bunch of lines collapsed into just:

(defn create-aim-proxy []
  (auto-proxy [com.aol.acc.AccEvents] []
     (OnImReceived [session imSession participant im] 
        (handle-im session imSession participant im))
     (OnStateChange [arg0 arg1 arg2]
        (handle-state-change arg0 arg1 arg2))))

Macros ftw. The nice thing about Clojure/Lisp is that it makes coding up this kind of reusable framework stuff really easy.

Clojure code density seems to be pretty good. There are a fair number of convenient shortforms in the language; for example, associative datatypes all act as a function — so given a hash map you can reference it with (my-hashmap :key). The base language itself is probably about as expressive as Python (or a bit better), but you have the added advantage of being able to use macros as needed to really get the code density up.

Nonetheless, I really wanted something like Ocaml’s / Haskell’s pattern matching; it makes some code wonderfully concise.

Accordingly, I hacked something up, based on Clojure’s built-in destructuring. Some examples:

Literal values match against the same value, while _ matches against any non-nil value (and nil matches against any nil one). Additionally, :when clauses can be used for conditional checks.

; simple recursive evaluator
(defn arithmetic [lst]
  (match lst
    v  :when (number? v)  v
    [ _ "error" _]     "error"
    [ _ _ "error"]     "error"
    [ "add" a b ]      (+ (arithmetic a) (arithmetic b))
    [ "sub" a b ]      (- (arithmetic a) (arithmetic b))
    [ "mul" a b ]      (* (arithmetic a) (arithmetic b))
    [ "div" a b ]      (/ (arithmetic a) (arithmetic b))
    [ "squared" a ]    (arithmetic ["mul" (arithmetic a) (arithmetic a)])
    _                  "error" ))

Both collections and single values can be used:

;; return signum fr a number
(defn signum [x]
  (match x 
     0 0
     n :when (< n 0) -1
     _ 1))

The pattern matching is stricter than the typical destructure; whereas [ a b ] will destructure against a list of any number of elements, [ a b ] will pattern match only against a list of two elements.

(match x 
    []    "empty"
    [_]   "one element"
    [a a] "two identical elements"
    [_ _] "two elements"
    _     "three or more")

If the same variable occurs in multiple locations in the parameter list, it will be checked for equality. The & tail form can be used to specify the rest of the list.

;; count identical elements in the same location in two lists:
(defn count= [ lst1 lst2 ]
  (loop [ a lst1 b lst2 count 0 ]
    (match [a b]
      [[e & at] [e & bt]]  (recur at bt (inc count))
      [[_ & at] [_ & bt]]  (recur at bt count)
      _                    count)))

Note that this is slightly more flexible than Haskell / ML, in that a variable of the same name can be multiple places in the pattern.

Defining

You can use the defnp macro to define a function that is pattern matched; it defines a function that takes one argument and has an implicit match statement. For example, the signum function can be written:

    (defnp signum
       0 0
       n :when (< n 0) -1
       _ 1)

(Thanks to Tom Faulhaber for suggesting this)

Gotchas

The Clojure destructuring will cause an exception if you try to destructure a collection type with a value.

(let [[a b] 10] a)
java.lang.UnsupportedOperationException: nth not supported on this type: Integer (NO_SOURCE_FILE:0)

… so be sure to check such cases early in your match statement, if they are possible.

How It Works

The pattern matcher uses the built-in Clojure destructuring as the main mechanism, but adorns it so that the pattern can be verified. For example, the code:

(match x [a a] "two identical" )

(let [ [ a g0001 & g0002 ] x ] 
     (if (and (not (nil? a)) (= g0001 a) (nil? g0002)) "two identical" nil))

That is, the destructuring is done, but then the two variables are checked to make sure that they are equal, and the list is checked to make sure it is only two elements long.

Source

Nonetheless, I ran into a situation where it seemed to be 100x slower than Java because of an erroneous assumption on my part. After stripping out and profiling and removing everything extraneous, it finally came down to array access. Ignoring the uselessness of this snippet for a while, the issue is how to translate something like the following code:

... which runs in about 350ms on my Macbook. My pass at the Clojure equivalent, with the type hinting:

;; this is 100x slower than the equivalent in Java
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]
        (aset-int v ix (unchecked-add (int 1) (aget v ix)))
        (aset-int v ix (unchecked-subtract (int 1) (aget v ix)))))))

... which takes about 40692ms -- more than 100x slower. What the hell? After many hours of Googling, I finally discovered from the comments on this blog post that aset-int is slower than aset, which I don't recall seeing anywhere else. Does it really make a difference?

;; much faster
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]>
        (aset v ix (unchecked-add (aget v ix) (int 1)))
        (aset v ix (unchecked-subtract (aget v ix) (int 1)))))))

... runs in about 1000ms, or about 3x slower than the Java variant, and is definitely something that I can live with. I had been assuming that aset-int was of course faster because it was more specific, but in fact (aset v ix (int val)) is much faster than (aset-int v ix val).

At any rate, I made a small, thin layer around the Tokyo Cabinet API, and put it on Github. Another thin wrapper can be found at this blog.

Copy of the README is below (the ultimate in lazy!).

Introduction

This is a simple interface to the Tokyo Cabinet libraries. Tokyo Cabinet is a very cool, very high performing key-value store. This library supports table mode, which essentially means that arbitrary hashmaps can be stored in the cabinet.

Note that this is appropriate for local storage only — if you’re looking to share a Tokyo Cabinet to multiple computers, you actually want Tokyo Tyrant.

Basic Usage

The with-cabinet call creates/opens a cabinet and allows the use of the various access routines within the scope of the call. For example, here’s how to create a cabinet with three entries.

(ns user (:use tokyo-cabinet))  ;; bring into our namespace
 
(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREAT) } 
    (doseq [[name val] [["1" "one"] ["2" "two"] ["3" "three"]]]
        (put-value name val)))

This creates a Tokyo Cabinet hash table, which allows one value per key. Now query an entry:

(with-cabinet { :filename "test.tokyo" :mode OREADER } 
    (get-value "1"))
"one"

Tables

A table in Tokyo Cabinet can be used to store arbitrary hash maps. For example:

(def params { :filename "test-table.tokyo" :mode (+ OWRITER OCREAT) :type :table } )
(with-cabinet params
    (put-value nil { :name "John Doe" :hobbies "rowing fishing skiing" :age 28 :gender "M" })
    (put-value nil { :name "Melissa Swift" :hobbies "soccer tennis books" :age 33 :gender "F"})
    (put-value nil { :name "Tom Swift" :hobbies "inventing exploring" :gender "M" })
    (put-value nil { :name "Harry Potter" :hobbies "magic quidditch flying" :gender "M" :age 9 }))

Queries

Queries can be run, and you can use (hint) to take a look at how the query is being performed:

; show a hint and all rows matching
(defn showrows [query]
    (let [showhint (atom false)] 
        (with-query-results row query
            (when (compare-and-set! showhint false true)
                  (println "Query: " query)
                  (println "Hint: " (hint))
                  (println "Results:"))
            (println row)))
        (println))
 
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Leads to the following output:

Query:  [[:age >= 30]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Indexes

Indexes can be added with create-index (and removed with delete-index), which help optimize particular queries.

The different index types:

• INDEX-DECIMAL
• INDEX-LEXICAL
• INDEX-QGRAM

With some optional specifiers that can be added / ored in:

• INDEX-KEEP — keep the index if it already exists
• INDEX-OPTIMIZE

Running the queries again, with indexes:

; indexes are persistent
(with-cabinet params
    (create-index :hobbies INDEX-TOKEN)
    (create-index :age INDEX-DECIMAL))
 
; try the queries again with the indexes in place
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Gets the following hint:

Query:  [[:age >= 30]]
Hint:  using an index: ":age" asc (NUMGT/NUMGE)
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  using an index: ":hobbies" inverted (STROR)
token occurrence: "soccer" 1
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Optional Search Parameters

You can further control what’s fetched by using a number of optional specifiers in the query:

• :limit nnn — limits the number of rows returned
• :skip nnn — skips the first nnn rows
• :sort fieldname — sorts by the given field
• :order val — the specific ordering, one of SORT-NUM-ASC, SORT-NUM-DESC, SORT-TEXT-ASC, or SORT-TEXT-DESC

For example:

(with-cabinet params (with-query-results row [] (println (:name row))))    
John Doe
Melissa Swift
Tom Swift
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name]] (println (:name row))))
Harry Potter
John Doe
Melissa Swift
Tom Swift
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC]] (println (:name row))))
Tom Swift
Melissa Swift
John Doe
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC] [:limit 1]] (println (:name row))))
Tom Swift

Lower Level

Depending on your application, it might not be convenient to have to bracket everything with with-cabinet, since that means an open and close of the cabinet. You can also use the lower level open-cabinet and close-cabinet calls, along with the “with” statement. This is also an easier way to use it at the command line. For example:

(def test-database (open-cabinet { :filename "test-open.tokyo" :mode (+ OWRITER OCREAT) }))
(with test-database (put-value "1" "one"))
(with test-database (get-value "1"))
(with test-database (print (primary-keys)))
(close-cabinet test-database)

Miscellaneous

Use (primary-keys) to return a lazy list of primary keys.

(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREATE) :type :table }
    (print (primary-keys)))

Links

• Tokyo Cabinet
• Tokyo Cabinet / Java API