Step 1: Install msmtp (or any other program)

On OS X you can do this with “brew install msmtp”. (You can use sendmail or whatever, but msmtp was easy to set up.).

Step 2: Set up msmtp

Set up the appropriate configuration file. I ran into a problem wherein the program would just hang; turning on debug showed that it hung after “Reading recipients from command line.” This turned out to be an incorrect SSL configuration — I needed tls_starttls to be off.

My particular .msmtprc configuration:

defaults
tls on
tls_starttls off
tls_certcheck off
 
account Work
host mail.host.com
domain domain.com
auth on
port 465
protocol smtp
from from-name
user user-name
password password

Step 3: install sendmail replacement

For me, this was two things: a small shell script and a Python program that will convert any plaintext e-mail into a plaintext + HTML e-mail.

Shell script mark_send:

#!/bin/sh
/Users/tim/mutt/mark_and_send.py "/usr/local/bin/pandoc -s" | /usr/local/bin/msmtp -a Work $@

Python script mark_and_send.py:

#!/usr/bin/python
import os
import sys
import subprocess
import email
import email.parser
import tempfile
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
 
if len(sys.argv) == 1:
    print "usage: mark_and_send.py markdown_program [markdown_flag]"
    print "where:"
    print "    markdown_program is the name of the program to translate markdown."
    print "    markdown_flag is an optional indicator in the first line that determines whether to run the markdown program"
    sys.exit(0)
 
markdown_program = sys.argv[1]
 
p = email.parser.Parser()
m = p.parsestr(sys.stdin.read())
 
# is this a single header?
do_markdown = False
if not m.is_multipart() and m.get_content_maintype() == 'text' and m.get_content_subtype() == 'plain':
    do_markdown = True
    payload = m.get_payload()
    # optionally check the markdown flag
    if len(sys.argv) == 3:
        lines = payload.split("\n")
        do_markdown = lines[0] == sys.argv[2]
        if do_markdown:
            payload = "\n".join(lines[1:])
 
if do_markdown:
    tf = tempfile.NamedTemporaryFile(delete=False)
    tf.write(payload)
    tf.close()
 
    (markdown, error) = subprocess.Popen(markdown_program.split() + [tf.name], stdout=subprocess.PIPE).communicate()
 
    # create a new message with the exact same headers
    new_message = MIMEMultipart('alternative')
    for (k,v) in m._headers:
        new_message[k] = v
 
    text_plain = MIMEText(payload, 'plain')
    new_message.attach(text_plain)
 
    text_html = MIMEText(markdown, 'html')
    new_message.attach(text_html)
 
    print new_message.as_string()
    os.unlink(tf.name)
else:
    print m.as_string()

The mark_and_send.py program takes one required argument (the name of the program to convert from markdown to HTML, possibly quoted) an an optional indicator of markdown. If the first line of the message is the indicator, then it is stripped and the rest of the message is converted to HTML.

Step 3: set up mutt

You need to tell mutt to use your shell script instead of sendmail. I have it inside a folder hook:

folder-hook +work/.* 'set sendmail="/bin/sh /Users/tim/mutt/mark_send"'

Now, whenever you send a plain-text email, mutt will automatically run markdown on it and attach it as the HTML view of the message.

Results

• Markdown is not ideal for e-mail, at least, not without some work. For example: more e-mails that are quoted by default will not work properly in markdown, because there are not blank lines in the message to set off the blockquote. This could possibly be improved by post-processing the file that Mutt generates on a reply, but I don’t think there’s a hook for that.
• Code coloring is nice
• Math formulas, if you need them

Example is show below, as rendered by Zimbra. It isn’t perfect. Google will unfortunately strip the colors from the code coloring, and Zimbra loses leading indents, and both readers will require the user to load images. Nonetheless, in a pinch…

• usb_buffer_free and usb_buffer_alloc had changed in the most recent kernel
• needed to add a device code for my adapter
• did not need to update the firmware or the USB list

• download files from the blog
• show differences
• copy changes wholesale from the blog to local, or vice versa
• add new posts

It’s all on Github.

Setting Up

Assuming that your blog is set up at http://www.yourblog.com (you can create an account at wordpress.com to test this out), all you’ll need to do is:

-- make a directory for the blog
mkdir blog
chdir blog

-- download everything
python wp.py --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php init
(wait a bit)

-- now set up so we don't have to specify --user, --password, and --url every time (optional)
python wp.py --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php defaults
-- you can skip specifying the password, and it'll prompt you when you run it

The files are downloaded in the appropriate YYYY/MM directories, with the draft directory being used for all of your unpublished drafts.

All the drafts are stored in plain text, but you’ll see some lines starting with periods — these are various WordPress variables that are associated with the file. You can change them, as well; for example, to change the title of the post, just change the line that begins with “.title”.

Seeing What’s Different

You can use the status command to see differences between the local file system and your blog.

python wp.py status

Note that only the most recent files are checked. If you want to really check every single file for changes, do:

python wp.py status all

You can also use the –diff command line option to see the differences between local and server:

python wp.py --diff status all

Updating From The Blog

If you’ve made changes through the web interface and you’d like to bring them down, you don’t have to download everything again, but can instead just update.

python wp.py pull

Again, only the most recent changes are brought down. If you want to check every post on the blog, do:

python wp.py pull all

Posting To The Blog

If you’ve made changes to files and you’d like to post them back, do:

python wp.py push

To push everything (and not just the most recent files), do:

python wp.py push all

Note that push only changes those files that exist in both spots. If you’re adding a new post, use the “post” command.

Posting/Editing

If you’d like to add a new post, put it in the drafts folder, and then do:

python wp.py post drafts/filename

Note that add can actually take existing posts, as well — it just forces an update of that one file, rather than running through all changes like push.

Publishing

To publish a file, just change the .post_status field from ‘draft’ to ‘published’. Note that doing this will cause a copy to move from the drafts folder to the appropriate year/month.

Gotchas

There are some gotchas due to the fact that the filename can change on you. There are cases where the filename that will be brought down is different then the one that you send up:

• You post a file without a .title or .wp_slug line
• You post a file with a different file name than the slug that is generated (i.e., “my-first-draft” when the title is actually “my final draft”)

The Google code provides a command line program that can generate secret keys as well as a PAM module, but it turns out to be very little code to authenticate a TOTP, thereby providing two-factor authentication to your website very easily.

To give the user the key, you’ll need to generate a cryptographically-secure 10 byte random key, presented to the user as a base32 16-character string. They can either enter this string directly, or you can use Google charts to provide a barcode that they can scan into the Google Authenticator application:

def get_barcode_image(username, domain, secretkey):
    url = "https://www.google.com/chart"
    url += "?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/"
    url += username + "@" + domain + "%3Fsecret%3D" + secretkey
    return url

For an example of what a code looks like, click here, or, look below:

After the user has a secret key from you and has entered it into Google Authenticator either by typing it in directly or scanning in the barcode, you have to be able to verify the key during login (for example). The code to authenticate is only a few lines in Python:

import time
import struct
import hmac
import hashlib
import base64
 
def authenticate(secretkey, code_attempt):
    tm = int(time.time() / 30)
 
    secretkey = base64.b32decode(secretkey)
 
    # try 30 seconds behind and ahead as well
    for ix in [-1, 0, 1]:
        # convert timestamp to raw bytes
        b = struct.pack(">q", tm + ix)
 
        # generate HMAC-SHA1 from timestamp based on secret key
        hm = hmac.HMAC(secretkey, b, hashlib.sha1).digest()
 
        # extract 4 bytes from digest based on LSB
        offset = ord(hm[-1]) & 0x0F
        truncatedHash = hm[offset:offset+4]
 
        # get the code from it
        code = struct.unpack(">L", truncatedHash)[0]
        code &= 0x7FFFFFFF;
        code %= 1000000;
 
        if ("%06d" % code) == str(code_attempt):
            return True
 
    return False

I wanted an easy way of running Clojure programs under Hadoop, and ended up writing a silly, simple little shim that would simply take a Clojure file with a mapper and reducer function. It means that there is no .JAR building and no AOT compiling — just write your mapper and reducer function, bundle up the file, and go. Note that the JAR is built for Hadoop 0.20 and later.

Getting The Jar

The quickest way of downloading the .JAR is to just download it.

Building The Jar

Build the shim:

javac -cp /usr/lib/hadoop/hadoop-core.jar:/usr/lib/hadoop/lib/*:lib/* -d classes Shim.java

Create a lib directory and add the clojure classes:

mkdir lib
cp /from/wherever/clojure-1.2.0.jar lib
cp /from/wherever/clojure-contrib-1.2.0.jar lib

Bundle it together into one .JAR file:

jar -cvf shim.jar -C classes/ . lib/*

[~/github/hadoop-shim] $ jar tf shim.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/brool/
com/brool/Shim.class
com/brool/Shim$Reduce.class
com/brool/Shim$Map.class
lib/clojure-1.2.0.jar
lib/clojure-contrib-1.2.0.jar

Using The Jar

To run the example wordcount:

hadoop jar shim.jar com.brool.Shim -files wordcount.clj input-file output-file

The output from the run will be in output-file/part-r-00000

More Details

The Clojure file that you provide should be in the user namespace, and must provide two functions named mapper and reducer.

Mapper

The mapper function takes a string representing one line in the input file and returns a list of [ key value ] pairs. A given input line can generate any number of map lines.

Reducer

The reducer is given a key and all values that were associated with that key. The reducer’s function is to consolidate all of that into one output line.

Example

Since word count is the canonical example, let’s do that. The mapper for word count is:

(defn mapper [v]
  (let [words (enumeration-seq (StringTokenizer. v))]
    (map #(do [ % "1" ] ) words)
))

As an example, for an input line of “This is a test, is it not?”, the following map will be generated:

(["This" "1"] ["is" "1"] ["a" "1"] ["test," "1"] ["is" "1"] ["it" "1"] ["not?" "1"])

Before being given the reducer, the Hadoop system will group all like keys together, resulting in:

"This" => [ "1" ]
"is"     => [ "1" "1" ]

So on and so forth. The reducer is given the key and the list of all values that had that key, and then emits the final result — for a word count, the correct code would be:

(defn reducer [k v]
  [ k (reduce + (map #(Integer/parseInt %) v)) ])

This simply adds up the counts.

Testing

In wordcount.clj there are two handy functions that allow for debugging of the mapper and reducer before submitting it to Hadoop.

Given a local file, the test-mapper will load the file and run it through the mapper; if your file is large you may just want to (take 20 (test-mapper “/my/filename”)).

The test-reducer function will load the file and run it through both the mapper and the reducer. Taking the example sentence above:

user> (test-reducer "/tmp/one-sentence")
(["This" 1] ["a" 1] ["is" 2] ["it" 1] ["not?" 1] ["test," 1])

Example #2

As another example: let’s say that you have a collection of log entries, and would like to record the first and last log entry for every user. Assume that the files are in a CSV format, with the fields being in the order of timehit, userid. Example:

2010-10-04 13:04:22,112334
2010-10-04 10:04:22,182994
2010-10-04 10:05:18,182994
2010-10-04 10:07:19,182994
2010-10-04 13:28:41,112334
2010-10-04 10:09:22,182994
2010-10-04 13:56:22,112334
2010-10-04 11:30:01,182994

The mapper for this:

(defn mapper [v]
    (let [[timehit userid] (.split v ",")]
        [ [ userid timehit ] ]
))

The reducer:

(defn reducer [k v]
(let [s (sort v)]
[k (str (first s) "," (last s))]))

The source is all on Github.

Beautiful Soup is a really nice Python library for extracting content from possibly-sloppy HTML, and I wanted some reasonably close Clojure equivalent. Unfortunately, the standard classes don’t work well with malformed HTML; as an example:

    => (require '(clojure [xml :as xml]))
    => (xml/parse "http://www.google.com")
    org.xml.sax.SAXParseException: The markup in the document preceding the root element must be well-formed. (NO_SOURCE_FILE:0)

Fortunately, there is already a TagSoup library that can parse non-perfect HTML, and it is very easy to integrate TagSoup into xml/parse. This module hardly does anything; it simply adds a few helper routines and brings the most-used calls into one amazingly bad namespace name.

Examples

Building your soup:

    (use beaujiful-soup.core)
 
    ; build soup from URL
    (def t (build-soup "http://www.google.com"))
 
    ; build soup from (deliberately malformed) string
    (def t2 (build-string-soup "<html><body><ul><li>One<li>Two</ul></body></html>"))

Extracting information is done with the xml-> call. Oftentimes the last thing you do will be a node or text or (attr :attribute) call, in order to convert the results into a more workable type:

    ; you can "walk" down the tree with successive tag names.  For
    ; example, get every list item inside the unordered list
    ; immediately inside the body.
    (xml-> t2 :body :ul :li node)
    ; => ({:tag :li, :attrs nil, :content ["One"]} {:tag :li, :attrs nil, :content ["Two"]})
 
    ; get the text for the list items
    (xml-> t2 :body :ul :li text)
    ; => ("One" "Two")
 
    ; Get textareas immediately inside the body.
    (xml-> t :body :textarea node)
    ; => ({:tag :textarea, :attrs {:id "csi", :style "display:none"}, :content nil})
 
    ; use descendants to iterate through all nodes, not just the immediate children.
    ; Get the text from all <a> tags anywhere in the body.
    (xml-> t descendants :a text)
    ; => ("Images" "Videos" "Maps" ...)
 
    ;  Get the href attribute from all tags
    (xml-> t descendants :a (attr :href))
    ; => ("http://www.google.com/imghp?hl=en&tab=wi" ... )

Use the (attr=) predicate to match an attribute value:

    ; find invisible stuff
    (xml-> t2 descendants (attr= :style "display:none") tag)
    ; => (:textarea :iframe)

Strings match the text inside nodes:

    ; find the link for the <a> that has "Videos" for content
    (xml-> t descendants :a "Videos" (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Arbitrary predicates can be used as well. They will take a loc (location), and are usually converted to a node before being used:

    ; find any :p or :div
    (defn p-or-div [loc] (contains? #{:p :div} (:tag (node loc))))
    (xml-> t descendants p-or-div tag)
    ; => (:div :div :div :div :div :div :div :div :div :div :div :p :div :div)
 
    ; find the link for <a> that has case-insensitive "Videos" for content
    (require 'clojure.string)
    (defn f [loc] 
      (let [n (node loc)]
       (and (= (:tag n) :a) (= (clojure.string/upper-case (first (:content n))) "VIDEOS"))))
    (xml-> t descendants f (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Fundamentally, the xml-> call returns a list of locations, and you can apply arbitrary transforms as necessary. For example, let’s say that you want to build a map of text => hrefs for all of the links:

    (defn loc-to-pair [loc]
        [ (attr loc :href), (text loc) ])
    (apply hash-map (xml-> t descendants :a loc-to-pair))
    ; => {"/services/" "Business Solutions",  ... }

Having a vector in the chain applies all the predicates within the vector, and filters out anything that doesn’t match. It acts a little like a lookahead in a regex. For example:

    ; Find the IDs of all divs that contain an href immediately within them
    (xml-> t descendants :div [ :a ] (attr :id))
    ; => ("fll")
 
    ; Find the IDs of all divs that contains an href anywhere within them
    (xml-> t descendants :div [ descendants :a ] (attr :id))
    ; => ("ghead" "gbar" "guser" "fll")

Source

(defproject mydirectory "1.0.0-SNAPSHOT" :description "FIXME: write" 
  :dependencies [[org.clojure/clojure "1.1.0-alpha-SNAPSHOT"] 
    [org.clojure/clojure-contrib "1.0-SNAPSHOT"]
    [mysql/mysql-connector-java "5.1.6"]
    [incanter/incanter "1.0-master-SNAPSHOT"]])

(that is, add the mysql and incanter dependencies).

Download all the dependencies with lein deps

Start up a REPL with everything in the classpath by just using lein repl

Wow, that’s kind of nice.

Unfortunately, despite the fact that there were some good guides on how to do it, it turned out there were a few gotchas, and it took significantly longer than 10 minutes. So, in the hopes that I can save someone else the time:

Use this guide, but:

• in Windows 7, you’ll want to use the “Network and Sharing Center” (right click on your network icon) to create a new “computer-to-computer” network. This is the one the iPhone will connect to.
• When you configure Putty, you’ll want to make sure you go to the SSH/Tunnels page and a) add a dynamic tunnel to a local port (assume 8080) — you won’t need to specify a destination and can leave that blank, and b) go to the “Connection” page and specify a keepalive of 5 or so. This keeps the SSH connection between the computer and the iPhone from dying every 30 seconds.
• You cannot use Chrome right now, because it does not properly forward DNS requests to the proxy (it took me hours to figure this out). Use Firefox and specify a SOCKS v5 proxy at port 8080 (or whatever) for the proxy, but you’ll also need to make sure to go to the about:config page and change the network.proxy.socks_remote_dns to “true”.
• Doing things in a particular order makes this work better. If the 3G connection on the iPhone is not active, then there doesn’t seem to be a way to activate after it links to your ad hoc connection. So: turn off wireless, go to a web page (this activates the 3G), turn on wireless, connect to your computer. This way both the 3G and wireless connections are established. An easy way to see if the 3G is active is to run MobileTerminal (or just SSH into the iPhone) and run ipconfig; if the pdp_ip0 device is pointing to a real IP, then your 3G is active.

I really enjoy Clojure. Everything seems so well thought out and well designed; in a lot of ways it reminds me of Python, which is three ways to ironic because back in the day I started using Python because it was very Lispy. The cycle of life turns even on computer languages.

But unfortunately, I get the sense that there’s probably only room for one JVM language to make it big. As much as I like Clojure, I think that Scala will probably be the one that succeeds. It’s got the static typing, it’s got the Java-like performance, and it’s got a syntax that’s close-ish to Java, so it fills the niche that Java fills so well: a programming language that is ideal for large groups of developers.

However, it’s early enough in the development of Clojure that there’s the sense that maybe individual efforts could make a difference. (As a counterexample, Haskell is too far along — it seems like everything has been done already, or anything that needs to be done requires a deep knowledge of category theory and the latest functional design papers.)

Assuming that I would want to help out Clojure, what could be done?

Donate? Actually, I did that. I figured that I got as many hours of enjoyment out of Clojure as I did out of a typical game, so I contributed the price of a game.

Libraries? Well, I’ve written one or two. Will probably write more.

If Clojure is going to succeed, it needs a niche, an area in which it performs so well that it blows everything else away. Concurrent programming is one possibility, but there is a lot of competition there.

I have a modest proposal:

Clojure should be more like PHP.

Yes, yes, I know that programming in PHP is socially reprehensible, just one step above people that shop without their shirt while in Walmart and car salesmen.

But what if we were to give Clojure the trappings that make starting developers try it? Metaphorically, it’s like Gulliver’s Travels; it looks like a humorous, light-hearted adventure story, but underneath it’s biting political satire and deep thoughts. We lure developers in with the possibility of an easy, light-hearted way to write web sites… and then once they’re in there, once they’ve progressed from manipulating web sites and start to realize the benefits of advanced languages, they can deal with a sane language, built on top of the JVM and with the full power of a Lisp.

So call this Gulliver.

Why Did PHP Succeed?

My impression, based on absolutely no citations or hard evidence, is that PHP became popular primarily because it is:

• Easy to start trying — start with an HTML page, and just add one statement to start playing around with PHP itself
• Easy to access data — typically, with MySQL
• Easy to deploy — just upload a file and go

So, continuing the gedankenexperiment: if we were designing something that would be very approachable and very easy to use, what would it look like?

Making Compojure More Approachable

Compojure is an excellent base to start with, but it strays away from templates, and instead tries to do everything with compojure.html, a library that allows you to form everything with vectors and lists:

(html [:p "2 + 2 = " (+ 2 2)])

Handy? Yes, if you’re a developer. But if a web designer is doing your HTML, or if you’re just starting out programming and you already know HTML, though, it seems an additional barrier to entry. So what if we turn it inside out and do it like PHP does it?

<p>2 + 2 = <?= (+ 2 2) ?>

Trying something more complicated, in compojure.html this:

(html [:ul (map (fn [x] [:li x]) (range 10))])

<ul>
  <? (dotimes [x 10] ?>
  <li><?= x ?>
  <? ) ?>
</ul>

A little bit longer, but treating it as HTML makes it conceptually easier for beginning programmers to understand, especially if you want to do something like changing the id or class with every row so your CSS can make it all pretty:

<ul>
  <? (dotimes [x 10] ?>
  <li id="list-<?= x ?>" class="<?= (if (even? x) "even" "odd") ?>"><?= x ?></li>
  <? ) ?>
</ul>

Doing this is compojure.html is trickier:

(html 
  [:ul 
    (map (fn [x] [:li {:id (str "list-" x) :class (if (even? x) "even" "odd")} x]) 
    (range 10))])

So, let’s lure people in with easy to use, and they can pick up vectors and S-exprs and notational convenience later. Converting a template file into the basic code is easy:

(defn- transform-string
  "Internal.  Transform a string into the appropriate echo statement.
An empty string translates into a single space."  
  [s]
  (if (> (.length s) 0)
    (format "(gulliver/echo %s)" (pr-str s))
  " "))
 
(defn- transform-code
  "Internal.  Transform code so that it can run as a render routine."
  [s]
  (if (= (first s) \=)
    (format "(gulliver/echo %s)" (.substring s 1))
    s))
 
(defn- convert* 
  "Internal.  Given a string that represents a Clojure template,
produce a flat list of the statements necessary to produce that template."
  [s]
  (let [m (re-matcher #"(?s)(.*?)<\?(.+?)\?>|(.+?)$" s)]
    (loop [next (re-find m) buffer ["'("]]
      (cond
       (not (nil? (nth next 3))) 
           (eval (read-string (apply str (conj buffer (transform-string (nth next 3)) ")" ))))
       :otherwise (recur 
                   (re-find m) 
                   (conj buffer 
                         (transform-string (nth next 1)) 
                         (transform-code (nth next 2))))
))))

Resulting in…

==> (convert* "<ul><? (dotimes [x 10] ?><li><?= x ?></li><? ) ?></ul>")
 
((gulliver/echo "<ul>") (dotimes [x 10] (gulliver/echo "<li>") (gulliver/echo x) (gulliver/echo "</li>")) (gulliver/echo "</ul>"))

Once you have the basic code generation, it’s easy enough to hook it up to a default servlet that automatically invokes it when necessary. There’s a fair bit of handwaving going on here — regexes are questionable for this, there are some interesting issues with namespaces, and it would require some method of resolving dependencies between files, but heck, it’s only 100 lines of code; it was really quick to put together and gives us a sense of whether this can work out.

So, with the source on github, your basic server becomes…

(use 'gulliver)
(defserver web-server
  {:port 8080}
  "/*"  (servlet gulliver/template-servlet))

Easy! As an test, an example file:

<?
(import 'java.util.Date)
?>
 
<html>
  <body>
    <h1>Delivered via Gulliver</h1>
    <p>Expressions at work: <?= (+ 2 2) ?></p>
    <p>Looping:</p>
    <ul>
    <? (dotimes [x 10] ?>
      <li><?= x ?></li>
    <? ) ?>
    </ul>
 
    <p>Here's a date:  <?= (Date.) ?></p>
  </body>
</html>

… does the right thing. How about a time test?

<html>
    <body>
    <h1>Starting time test...</h1>
    <? (let [ start (System/currentTimeMillis) ] ?>
    <p>Adding 1..100000:  <?= (reduce + 0 (range 1 1000001)) ?></p>
    <p>(took <?= (- (System/currentTimeMillis) start) ?> ms)</p>
    <? ) ?> 
    </body>
</html>

… runs in about 50ms. PHP takes about 130ms on the same sequence on my laptop, so, while there are tons of optimizations we could do (like not doing a stat on the file every time the page is rendered (!)), it nonetheless performs well enough that maybe this is a viable option. Finally, as a special bonus, a stupid eval trick that evaluates Clojure code:

<html>
    <body>
        <? (when ((request :params) :code) ?>
        <pre style="border: 1px dotted">
        <?= (str (eval (read-string ((request :params) :code)))) ?>
        </pre>
        <? ) ?>
        <form method="post">
        <textarea name="code" rows=10 cols=80></textarea><br/>
        <input type="submit" value="Evaluate" />
        </form>
    </body>
</html>

Take that, non-dynamic languages!

Github copy of the source

Next part: Schema-ing Against MySQL

public void OnIdleStateChange(AccSession arg0, int arg1) {
}
 
public void OnInstanceChange(AccSession arg0, AccInstance arg1, AccInstance arg2, AccInstanceProp arg3) {
}
 
public void OnLookupUsersResult(AccSession arg0, String[] arg1, int arg2, AccResult arg3, AccUser[] arg4) {
}
 
public void OnSearchDirectoryResult(AccSession arg0, int arg1, AccResult arg2, AccDirEntry arg3) {
}
 
// ... go on like this for pages

The Java code is here, if you’re interested in the entire set of calls. Now, I didn’t care about most of those events, but I had to override them, since they didn’t have a default implementation. What made this seem painful was that I was really only interested in two of the callbacks. So I started to record an Emacs macro to convert the Java code to the equivalent Clojure proxy statement, and then I realized that I didn’t have to — I was using a Lisp.

(defmacro auto-proxy [interfaces variables & args]
  (let [defined (set (map #(str (first %)) args))
        names (fn [i] (map #(.getName %) (.getMethods i)))
        all-names (into #{} (apply concat (map names (map resolve interfaces))))
        undefined (difference all-names defined) 
        auto-gen (map (fn [x] `(~(symbol x) [& ~'args])) undefined)]
    `(proxy ~interfaces ~variables ~@args ~@auto-gen)))

Auto-proxy works just like proxy, but it makes an empty implementation for any call that wasn’t defined. So, suddenly, what would have been a bunch of lines collapsed into just:

(defn create-aim-proxy []
  (auto-proxy [com.aol.acc.AccEvents] []
     (OnImReceived [session imSession participant im] 
        (handle-im session imSession participant im))
     (OnStateChange [arg0 arg1 arg2]
        (handle-state-change arg0 arg1 arg2))))

Macros ftw. The nice thing about Clojure/Lisp is that it makes coding up this kind of reusable framework stuff really easy.

Clojure code density seems to be pretty good. There are a fair number of convenient shortforms in the language; for example, associative datatypes all act as a function — so given a hash map you can reference it with (my-hashmap :key). The base language itself is probably about as expressive as Python (or a bit better), but you have the added advantage of being able to use macros as needed to really get the code density up.

Nonetheless, I really wanted something like Ocaml’s / Haskell’s pattern matching; it makes some code wonderfully concise.

Accordingly, I hacked something up, based on Clojure’s built-in destructuring. Some examples:

Literal values match against the same value, while _ matches against any non-nil value (and nil matches against any nil one). Additionally, :when clauses can be used for conditional checks.

; simple recursive evaluator
(defn arithmetic [lst]
  (match lst
    v  :when (number? v)  v
    [ _ "error" _]     "error"
    [ _ _ "error"]     "error"
    [ "add" a b ]      (+ (arithmetic a) (arithmetic b))
    [ "sub" a b ]      (- (arithmetic a) (arithmetic b))
    [ "mul" a b ]      (* (arithmetic a) (arithmetic b))
    [ "div" a b ]      (/ (arithmetic a) (arithmetic b))
    [ "squared" a ]    (arithmetic ["mul" (arithmetic a) (arithmetic a)])
    _                  "error" ))

Both collections and single values can be used:

;; return signum fr a number
(defn signum [x]
  (match x 
     0 0
     n :when (< n 0) -1
     _ 1))

The pattern matching is stricter than the typical destructure; whereas [ a b ] will destructure against a list of any number of elements, [ a b ] will pattern match only against a list of two elements.

(match x 
    []    "empty"
    [_]   "one element"
    [a a] "two identical elements"
    [_ _] "two elements"
    _     "three or more")

If the same variable occurs in multiple locations in the parameter list, it will be checked for equality. The & tail form can be used to specify the rest of the list.

;; count identical elements in the same location in two lists:
(defn count= [ lst1 lst2 ]
  (loop [ a lst1 b lst2 count 0 ]
    (match [a b]
      [[e & at] [e & bt]]  (recur at bt (inc count))
      [[_ & at] [_ & bt]]  (recur at bt count)
      _                    count)))

Note that this is slightly more flexible than Haskell / ML, in that a variable of the same name can be multiple places in the pattern.

Defining

You can use the defnp macro to define a function that is pattern matched; it defines a function that takes one argument and has an implicit match statement. For example, the signum function can be written:

    (defnp signum
       0 0
       n :when (< n 0) -1
       _ 1)

(Thanks to Tom Faulhaber for suggesting this)

Gotchas

The Clojure destructuring will cause an exception if you try to destructure a collection type with a value.

(let [[a b] 10] a)
java.lang.UnsupportedOperationException: nth not supported on this type: Integer (NO_SOURCE_FILE:0)

… so be sure to check such cases early in your match statement, if they are possible.

How It Works

The pattern matcher uses the built-in Clojure destructuring as the main mechanism, but adorns it so that the pattern can be verified. For example, the code:

(match x [a a] "two identical" )

(let [ [ a g0001 & g0002 ] x ] 
     (if (and (not (nil? a)) (= g0001 a) (nil? g0002)) "two identical" nil))

That is, the destructuring is done, but then the two variables are checked to make sure that they are equal, and the list is checked to make sure it is only two elements long.

Source

Nonetheless, I ran into a situation where it seemed to be 100x slower than Java because of an erroneous assumption on my part. After stripping out and profiling and removing everything extraneous, it finally came down to array access. Ignoring the uselessness of this snippet for a while, the issue is how to translate something like the following code:

... which runs in about 350ms on my Macbook. My pass at the Clojure equivalent, with the type hinting:

;; this is 100x slower than the equivalent in Java
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]
        (aset-int v ix (unchecked-add (int 1) (aget v ix)))
        (aset-int v ix (unchecked-subtract (int 1) (aget v ix)))))))

... which takes about 40692ms -- more than 100x slower. What the hell? After many hours of Googling, I finally discovered from the comments on this blog post that aset-int is slower than aset, which I don't recall seeing anywhere else. Does it really make a difference?

;; much faster
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]>
        (aset v ix (unchecked-add (aget v ix) (int 1)))
        (aset v ix (unchecked-subtract (aget v ix) (int 1)))))))

... runs in about 1000ms, or about 3x slower than the Java variant, and is definitely something that I can live with. I had been assuming that aset-int was of course faster because it was more specific, but in fact (aset v ix (int val)) is much faster than (aset-int v ix val).

At any rate, I made a small, thin layer around the Tokyo Cabinet API, and put it on Github. Another thin wrapper can be found at this blog.

Copy of the README is below (the ultimate in lazy!).

Introduction

This is a simple interface to the Tokyo Cabinet libraries. Tokyo Cabinet is a very cool, very high performing key-value store. This library supports table mode, which essentially means that arbitrary hashmaps can be stored in the cabinet.

Note that this is appropriate for local storage only — if you’re looking to share a Tokyo Cabinet to multiple computers, you actually want Tokyo Tyrant.

Basic Usage

The with-cabinet call creates/opens a cabinet and allows the use of the various access routines within the scope of the call. For example, here’s how to create a cabinet with three entries.

(ns user (:use tokyo-cabinet))  ;; bring into our namespace
 
(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREAT) } 
    (doseq [[name val] [["1" "one"] ["2" "two"] ["3" "three"]]]
        (put-value name val)))

This creates a Tokyo Cabinet hash table, which allows one value per key. Now query an entry:

(with-cabinet { :filename "test.tokyo" :mode OREADER } 
    (get-value "1"))
"one"

Tables

A table in Tokyo Cabinet can be used to store arbitrary hash maps. For example:

(def params { :filename "test-table.tokyo" :mode (+ OWRITER OCREAT) :type :table } )
(with-cabinet params
    (put-value nil { :name "John Doe" :hobbies "rowing fishing skiing" :age 28 :gender "M" })
    (put-value nil { :name "Melissa Swift" :hobbies "soccer tennis books" :age 33 :gender "F"})
    (put-value nil { :name "Tom Swift" :hobbies "inventing exploring" :gender "M" })
    (put-value nil { :name "Harry Potter" :hobbies "magic quidditch flying" :gender "M" :age 9 }))

Queries

Queries can be run, and you can use (hint) to take a look at how the query is being performed:

; show a hint and all rows matching
(defn showrows [query]
    (let [showhint (atom false)] 
        (with-query-results row query
            (when (compare-and-set! showhint false true)
                  (println "Query: " query)
                  (println "Hint: " (hint))
                  (println "Results:"))
            (println row)))
        (println))
 
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Leads to the following output:

Query:  [[:age >= 30]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Indexes

Indexes can be added with create-index (and removed with delete-index), which help optimize particular queries.

The different index types:

• INDEX-DECIMAL
• INDEX-LEXICAL
• INDEX-QGRAM

With some optional specifiers that can be added / ored in:

• INDEX-KEEP — keep the index if it already exists
• INDEX-OPTIMIZE

Running the queries again, with indexes:

; indexes are persistent
(with-cabinet params
    (create-index :hobbies INDEX-TOKEN)
    (create-index :age INDEX-DECIMAL))
 
; try the queries again with the indexes in place
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Gets the following hint:

Query:  [[:age >= 30]]
Hint:  using an index: ":age" asc (NUMGT/NUMGE)
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  using an index: ":hobbies" inverted (STROR)
token occurrence: "soccer" 1
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Optional Search Parameters

You can further control what’s fetched by using a number of optional specifiers in the query:

• :limit nnn — limits the number of rows returned
• :skip nnn — skips the first nnn rows
• :sort fieldname — sorts by the given field
• :order val — the specific ordering, one of SORT-NUM-ASC, SORT-NUM-DESC, SORT-TEXT-ASC, or SORT-TEXT-DESC

For example:

(with-cabinet params (with-query-results row [] (println (:name row))))    
John Doe
Melissa Swift
Tom Swift
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name]] (println (:name row))))
Harry Potter
John Doe
Melissa Swift
Tom Swift
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC]] (println (:name row))))
Tom Swift
Melissa Swift
John Doe
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC] [:limit 1]] (println (:name row))))
Tom Swift

Lower Level

Depending on your application, it might not be convenient to have to bracket everything with with-cabinet, since that means an open and close of the cabinet. You can also use the lower level open-cabinet and close-cabinet calls, along with the “with” statement. This is also an easier way to use it at the command line. For example:

(def test-database (open-cabinet { :filename "test-open.tokyo" :mode (+ OWRITER OCREAT) }))
(with test-database (put-value "1" "one"))
(with test-database (get-value "1"))
(with test-database (print (primary-keys)))
(close-cabinet test-database)

Miscellaneous

Use (primary-keys) to return a lazy list of primary keys.

(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREATE) :type :table }
    (print (primary-keys)))

Links

• Tokyo Cabinet
• Tokyo Cabinet / Java API

So, I wrote a tiny little program to help facilitate using git and WordPress together through the XMLRPC API, and it’s helped me write… well, if not more than at least less painfully.

Setting Up

Assuming that your blog is set up at http://www.yourblog.com (you can create an account at wordpress.com to test this out), all you’ll need to do is:

-- make a directory for the blog
mkdir blog
chdir blog
 
-- download everything
python wp.py download --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php
(wait a bit)
git init
git add .
git commit -m "first version"
 
-- now set up so we don't have to specify --user, --password, and --url every time (optional)
git wp config wp.url http://www.yourblog.com/xmlrpc.php
git wp config wp.user yourname
git wp config wp.password yourpass

The files are downloaded in the appropriate YYYY/MM directories, with the draft directory being used for all of your unpublished drafts.

All the drafts are stored in plain text, but you’ll see some lines starting with periods — these are various WordPress variables that are associated with the file. You can change them, as well; for example, to change the title of the post, just change the line that begins with “.title”.

Seeing What’s Different

You can use the status command to see differences between the local file system and your blog.

python wp.py status

Updating From The Blog

If you’ve made changes through the web interface and you’d like to bring them down, you don’t have to download everything again, but can instead just update.

python wp.py update

Posting/Editing

If you’d like to edit a post, just edit it, and then use wp post to push it back to the blog:

python wp.py post changed-file

To create a new post, just create a new file in the draft folder — I like to specify the .title and .wp_slug parameters, as well — and then post it. You can also publish it by changing the .post_status line from draft to publish.

Gotchas

While this program requires git, it doesn’t automatically check anything in — so you’ll need to make sure you do git add / git commit or whatnot as necessary.

There are also some gotchas due to the fact that the filename can change on you. There are cases where the filename that will be brought down is different then the one that you send up:

• You post a file without a .title or .wp_slug line
• You post a file with a different file name than the slug that is generated (i.e., “my-first-draft” when the title is actually “my final draft”)
• Something moves from “draft” to “publish”

When you see a message of the form “changed: fn1 -> fn2″, it means that a rename has occurred, and you’ll need to do the git rm/git add or git mv by hand.

Source

I needed to interface with a bunch of data that had PHP wrapper classes, and needed a quick way of being able to interface with PHP from Python. (At this point, you might find it hard to believe that the PHP wrapper classes were worth trying to reuse, but the wrapper classes took care of partitioning and caching and everything, so it seemed silly to not reuse them). I considered using the PHP API and ctypes to make a Python to PHP bridge , but then decided that was entirely too much work, so I just hacked together a stupid class that invoked PHP with arbitrary code and returned it in a number of ways.

Basic usage

Create the PHP class (giving it an optional prefix and postfix — prefix is the most useful, as it allows you to specify requires), and then invoke a code block with get_raw, get, or get_one. Note that every “call” into PHP will create and destroy a PHP process.

Some examples:

Reading raw input

php = PHP("require '../code/private/common.php';")
code = """for ($i = 1; $i <= 10; $i++) { echo "$i\n"; }"""
print php.get_raw(code)

will output a string containing the numbers “1″..”10″

Reading one big JSON value

php = PHP("require '../code/private/common.php';")
code = """                                                                                          
$a = array();                                                                                       
for ($i = 1; $i <= 10; $i++) { $a[] = $i; }                                                         
echo json_encode($a);"""
print php.get(code)

This code would return a Python list with the numbers 1..10.

Reading many JSON values

    php = PHP("require '../code/private/common.php';")
    code = """                                                                                          
    for ($i = 1; $i <= 10; $i++) {                                                                      
        echo json_encode(array($i => $i * $i)) . "\n";                                                  
    }"""
    for row in php.get_one(code):
        print row

Given a PHP snippet that returns one JSON value per line, iterate through them as Python values.

Find it on Github.

data S = S { name :: String } deriving (Show)
firstname s = (words (name s))!!0
lastname  s = (words (name s))!!1

But, dammit, you want to invoke it like you would in C++. So define a function:

*Main> let (-->) x f = f x
*Main> let test = S "George Washington"
*Main> test --> firstname
"George"
*Main> test --> lastname
"Washington"

(It’s tempting to use `.`, but it conflicts with the Prelude. Also note that you could define it as “(–>) = flip ($)”). But what if it takes more than one parameter?

*Main> let flip_concat t s = intercalate t $ reverse.words $ name s
*Main> test --> flip_concat ", "
"Washington, George"
*Main> test --> flip_concat(", ")
"Washington, George"

… although you might not like that anything can be applied to (–>).

*Main> [1,2,3] --> length
3
 
class Deref a where
    (-->) :: a -> (a -> b) -> b
    x --> f = f x 
 
instance Deref S
 
*Main> [1,2,3] --> length
 
<interactive>:1:0:
    No instance for (Deref [t])
      arising from a use of `-->' at <interactive>:1:0-17
    Possible fix: add an instance declaration for (Deref [t])
    In the expression: [1, 2, 3] --> length
    In the definition of `it': it = [1, 2, 3] --> length

You can even use tuple passing to make it look even more like a typical call.

*Main> let pretty (pre, mid, post) s = pre ++ (firstname s) ++ mid ++ (lastname s) ++ post
*Main> test --> pretty ("<", ", ", ">")
"<George, Washington>"

… although that makes it harder to curry.

{-# OPTIONS -XBangPatterns #-}
import IO
import System
import Data.Word
import Data.Array.Unboxed
import Control.Monad.ST
import Data.Array.ST
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C
 
counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   -- collate character counts here
                   return arr
counts !line = runSTUArray (counts' line)
 
hit !line = let freq = counts line in freq!65 == 10
 
main =
    do args <- getArgs
       f <- openFile "wordlist" ReadMode
       text <- B.hGetContents f
       print $ length $ filter hit (C.lines text)

This is fast (0.07s on 300K test file, compiled with -O2 on a Macbook Pro), but there’s no actual collation going on. It suddenly becomes two magnitudes slower as soon as you start to do anything based on the line:

-- version 1 : 9.5 seconds
counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   readArray arr (B.head line) >>= \v -> writeArray arr (B.head line) (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

.. equals 9.54 seconds just when you collate the first character of the string, and led me to believe that ByteStrings were slow, especially since a constant change to the array was fast (0.07 seconds):

-- version 2 : 0.07 seconds
counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   readArray arr 0 >>= \v -> writeArray arr 0 (v+1)
                   return arr

(Here I’ll elide the hours of me messing around with profiling and whatnot to try and figure out why B.head was so slow, or how to make B.foldl’ update a state variable, or sprinkling strictness bangs all over the place, or the other tons of false trails that I went down.)

It turns out that the slow portion in all of this is the newArray, not the character collation itself, which can be seen in the profile if the array creation is moved into its own routine and we look at the profile:

-- version 3
-- compile with: ghc -package bytestring stuarray.hs -prof -auto-all -O2 -o stuarray.out
-- run with: ./stuarray.out +RTS -p 
initial_array = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   return arr
 
counts' !line = do arr <- initial_array
                   let ix = B.head line
                   readArray arr ix >>= \v -> writeArray arr ix (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

… and the relevant part of the profile:

MAIN                     MAIN                     1           0   0.0    0.0   100.0  100.0
 main                    Main                   240           1   0.7    0.3   100.0  100.0
  hit                    Main                   242      335075   0.0    0.0    99.3   99.7
   counts                Main                   243      335075   0.0    0.2    99.3   99.7
    counts'              Main                   244      335075   0.0    0.0    99.3   99.5
     initial_array       Main                   245      335075  99.3   99.5    99.3   99.5

Which brings up two interesting points:

• The Haskell optimizer was so smart that it was able to figure out that the code in version 2 built a constant array… and so only called the newArray once. (Confirmed with a profile).
• New array creation seems to be so slow that it dominates a benchmark that has file I/O.

The Ocaml version took about 0.720 seconds (Ocaml version below), compared to Haskell time of about 8.5 seconds. Reducing the array size to 128 in both cases reduced it to 0.112 seconds for Ocaml and 4.3 seconds in Haskell.

Note the discontinuity in Ocaml between 255 and 256 elements, which I find interesting. The nice people in #haskell suggested that I stop constructing/destructing the array and instead just null it out between each line, and it turned out that the best way to do that was just to thaw a starter array.

-- version 4: 0.11 seconds
initial_array = listArray (0,255) (repeat 0) :: UArray Word8 Int
 
counts' !line = do arr <- thaw initial_array
                   let ix = B.head line
                   readArray arr ix >>= \v -> writeArray arr ix (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

This is still kind of strange to me (because an object is still getting constructed/destructed — thaw guarantees a copy) but you can’t argue with a performance increase. (Interestingly, using Array.copy or Array.fill in Ocaml is slower than just using Array.make). Final times? 0.11 seconds in Haskell vs. the best of 0.720 second UPDATED: 0.12 seconds in Ocaml… and the Haskell version is just 50% slower than a C implementation with a gratuitous calloc instead of a memset.

UPDATE: An pointed out that zeroing the Ocaml array with a for loop is much faster than Array.copy or Array.fill, bringing it to about the same speed as Haskell.

Lessons learned?

• Performance is a treacherous mistress
• The Haskell optimizer is awesome but you have to be wary when trying to narrow down performance problems
• #haskell is always full of useful suggestions
• Array creation seems to be slow enough that alternatives should be explored.

open Char
open String
 
let all_lines fn filename = 
    let chan = open_in filename in
        try
            while true do 
                let line = input_line chan in 
                    fn line
            done
        with End_of_file ->
            close_in chan
 
let initial_array = Array.make 256 0
 
let count line = 
    let freq = initial_array in
    let ix = int_of_char line.[0] in 
        for i = 0 to 255 do freq.(i) <- 0 done;
        freq.(ix) <- freq.(ix) + 1;
        freq
 
let hit line = 
    let freq = count line in
        freq.(65) = 10
 
let _ = 
    let linecount = ref 0 in 
        all_lines (fun line -> if hit line then linecount := !linecount + 1 else ()) "wordlist";
        print_int !linecount

import IO
import System
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C
 
main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f 
          print $ length $ filter ((==) searchfor) (C.lines text)

import IO
import System
import Data.Char
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C
 
main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f 
          print $ length $ filter (\x -> (C.map toLower x) == searchfor) (C.lines text)

Suddenly, the ByteString version becomes about 30% slower than the naive version — 0.337 seconds vs. 0.251 seconds — and is even slower than the Python version. What the heck is going on here? Trying an empty map (i.e., C.map id x) resulted in something fast, so I’m suspecting that the lowercase function itself is slow.

Unfortunately, there doesn’t seem to be a lowercase available in ByteString; at the moment it seems that you need to set up your own ctype table and use that.

import IO
import System
import Data.Char
import Data.Word
import Data.Array.Unboxed
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C
 
ctype_lower = listArray (0,255) (map (BI.c2w . toLower) ['\0'..'\255']) :: UArray Word8 Word8
lowercase = B.map (\x -> ctype_lower!x)
 
main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f 
          print $ length $ filter (\x -> (lowercase x) == searchfor) (C.lines text)

… which turns out to run really quickly at 0.070 seconds, about the same as a C program doing the same task.

Update: See dons comments below — Char is operating on Unicode, which makes it slow. I wonder if a ctype.h-type library for ByteString makes sense?