However, I just installed mu / mu4e and it works really well — well enough that I prefer reading e-mail in Emacs now, as opposed to just logging into Google mail and reading it there. I ended up archiving about 10 years of mail (approximately 10k of messages) off Gmail and mu can re-index it in seconds.

Installing offlineimap

I used offlineimap to sync everything with Gmail. Steal this configuration and put it in ~/.offlineimaprc (or whatever — but don’t name it .offlineimap, because that will cause errors). For my laptop I added:

maxage = 5
autorefresh = 15

… to keep only 5 days of mail and to automatically update it every 15 minutes.

Installing mu4e

The version of mu4e available in Ubuntu 12.04LTS is really old and throws spurious errors, so you’ll need to build your own. Get mu from github and then rebuild. I needed a few extra libraries in my installation.

sudo apt-get install libglib2.0-dev libgmime-2.6-dev libxapian-dev
sudo apt-get install texinfo
autoreconf -i

Installing into Emacs

Change into mu4e, do a make / sudo make install. I used pretty much just the longer configuration in the documentation, although I put everything into a deferred load (code below), and there are the following caveats:

• Viewing a message in a browser is handy, so I took the code from the Emacs Wiki.
• I needed to override mu4e-html2text-command because about 10% of HTML messages weren’t rendering.
• The documentation says to set mu4e-view-show-images to show images, but it’s actually mu4e-show-images.

(defun email () 
  (interactive)
 
  (when (not (featurep 'mu4e))
    (add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e/")
 
    (require 'mu4e)
    (require 'org-mu4e)
 
    ;; defaults
 
    (setq mu4e-maildir "~/Maildir")
    (setq mu4e-drafts-folder "/gmail/[Gmail].Drafts")
    (setq mu4e-sent-folder   "/gmail/[Gmail].Sent Mail")
    (setq mu4e-trash-folder  "/gmail/[Gmail].Trash")
 
    ;; don't save message to Sent Messages, Gmail/IMAP takes care of this
    (setq mu4e-sent-messages-behavior 'delete)
 
    ;; setup some handy shortcuts
    ;; you can quickly switch to your Inbox -- press ``ji''
    ;; then, when you want archive some messages, move them to
    ;; the 'All Mail' folder by pressing ``ma''.
 
    (setq mu4e-maildir-shortcuts
          '( ("/gmail/INBOX"               . ?i)
             ("/gmail/[Gmail].IMPORTANT"   . ?!)
             ;; ("/gmail/[Gmail].Sent Mail"   . ?s)
             ;; ("/gmail/[Gmail].Trash"       . ?t)
             ("/gmail/[Gmail].All Mail"    . ?a)))
 
    ;; allow for updating mail using 'U' in the main view:
    ;; I have this running in the background anyway
    ;; (setq mu4e-get-mail-command "offlineimap")
 
    ;; something about ourselves
    (setq
     user-mail-address "user@domain"
     user-full-name  "username"
     message-signature nil)
 
    ;; sending mail -- replace USERNAME with your gmail username
    ;; also, make sure the gnutls command line utils are installed
    ;; package 'gnutls-bin' in Debian/Ubuntu
 
    (require 'smtpmail)
 
    (if (string< emacs-version "24")
        (setq message-send-mail-function 'smtpmail-send-it
              starttls-use-gnutls t
              smtpmail-starttls-credentials '(("smtp.gmail.com" 587 nil nil))
              smtpmail-auth-credentials
              '(("smtp.gmail.com" 587 "user@domain" nil))
              smtpmail-default-smtp-server "smtp.gmail.com"
              smtpmail-smtp-server "smtp.gmail.com"
              smtpmail-smtp-service 587)
 
      ;; alternatively, for emacs-24 you can use:
      (setq message-send-mail-function 'smtpmail-send-it
            smtpmail-stream-type 'starttls
            smtpmail-default-smtp-server "smtp.gmail.com"
            smtpmail-smtp-server "smtp.gmail.com"
            smtpmail-smtp-service 587)
      )
 
 
    ;; don't keep message buffers around
    (setq message-kill-buffer-on-exit t)
 
    ;; show images
    (setq mu4e-show-images t)
 
    ;; use imagemagick, if available
    (when (fboundp 'imagemagick-register-types)
      (imagemagick-register-types))
 
    ;;; message view action
    (defun mu4e-msgv-action-view-in-browser (msg)
      "View the body of the message in a web browser."
      (interactive)
      (let ((html (mu4e-msg-field (mu4e-message-at-point t) :body-html))
            (tmpfile (format "%s/%d.html" temporary-file-directory (random))))
        (unless html (error "No html part for this message"))
        (with-temp-file tmpfile
          (insert
           "<html>"
           "<head><meta http-equiv=\"content-type\""
           "content=\"text/html;charset=UTF-8\">"
           html))
        (browse-url (concat "file://" tmpfile))))
 
    (add-to-list 'mu4e-view-actions
                 '("View in browser" . mu4e-msgv-action-view-in-browser) t)
 
    ;; convert org mode to HTML automatically
    (setq org-mu4e-convert-to-html t)
 
    ;; need this to convert some e-mails properly
    (setq mu4e-html2text-command "html2text -utf8 -width 72")
  )
  (mu4e)
)
 
(defalias 'org-mail 'org-mu4e-compose-org-mode)

Handy Keys in mu4e

• F — to forward
• C — to compose
• O — opens an attachment in detail view
• aV — views mail in HTML mode
• ji, j!, ja — jump to mailbox. (See mu4e-maildir-shortcuts above).
• mi, m!, ma — move message to mailbox
• V — turn on/off duplicates (handy if you’re looking at All Mail)
• W — turn on/off related messages
• / — narrow the current view via search
• ?, d, m — mark for unread / delete / move
• x — execute deferred marks

Org Mode

One of the neat things about mu4e is the fact that you can send messages built in org-mode, although the current implementation is a little flaky. Unless you do things the right way you’ll often get “Error 70,” and a plain unadorned e-mail will be sent out. Unfortunately, my friends and associates are not yet at that advanced stage of development that they’ll read org-mode format directly.

To use org-mode via compose, you:

• Compose a message
• Run M-x org-mu4e-compose-org-mode (hah, note my alias above)
• And, finally, and most important: you must send the e-mail via ^C ^S only when the cursor is in the header. If you try to send it while the cursor is in the body of the e-mail, it won’t work.

Yeah, not ideal. But hopefully fixable and it isn’t too hard in practice; only a small percentage of the e-mails I send out are rich text.

You may want to change the default formatting options; I found that having a Table of Contents in my e-mails was a bit too pretentious, and the formatting for code blocks left too much white space. Apply the following diff:

diff --git a/mu4e/org-mu4e.el b/mu4e/org-mu4e.el
index 44d5ea1..6697486 100644
--- a/mu4e/org-mu4e.el
+++ b/mu4e/org-mu4e.el
@@ -170,7 +170,9 @@ and images in a multipart/related part."
            ;; because we probably don't want to export a huge style file
            (org-export-htmlize-output-type 'inline-css)
            ;; makes the replies with ">"s look nicer
-           (org-export-preserve-breaks t)
+           (org-export-preserve-breaks nil)
+            ;; turn off table of contents
+            (org-export-with-toc nil)
            ;; dvipng for inline latex because MathJax doesn't work in mail
            (org-export-with-LaTeX-fragments 'dvipng)
            ;; to hold attachments for inline html images

Org Mode Example

And, okay, to be honest it’s not entirely useful in the day to day, but it is pretty neat that if you need to you can send an e-mail via:

Remember we were talking about functions of the form $f(x) = x^2$ the other day?  I wanted to show you what it looked like.
 
#+name: python-data
#+BEGIN_SRC python :exports both :results replace
return [ (ix, ix*ix) for ix in xrange(0,11) ]
#+END_SRC
 
It's hard to visualize this, though, so it helps to look at the graph.
 
#+BEGIN_SRC R :var dat=python-data :exports both :results output graphics :file graph.png
library(ggplot2)
ggplot(data=dat, aes(x=V1, y=V2))+geom_point()
#+END_SRC

… to get

It’s one of those things that makes me feel like I’m living in the future. I didn’t get my weekend trips to the moon, I didn’t get jetpacks, and I sure as heck didn’t get underwater cities, but at least the future has given us reasonable, electric cars.

The Leaf is not a bad little car. It’s actually much zippier than our Prius, as long as you aren’t driving in Eco mode, and handles fairly well. It’s odd driving down the freeway at 65 and not hearing an engine. Range is about 75 miles, give or take — and there’s a lot of give or take. Using the climate control, either heat or AC, will reduce mileage about 20%, and, annoyingly, the car defaults to having the climate control on, so you have to remember to turn it off.

The Leaf takes has a battery that takes 24KWh, so it works about to about 3 miles / KWh. When we leased (2012) there was a subsidy for the installation of a Blink charger.

The advantage of the Leaf is chiefly a guaranteed sticker in California for the carpool lane, but it’s also cheaper per mile. My girlfriend was paying about $100 / week in gas — so leasing the Leaf + $0.05 / mile turned out to be cheaper than gasoline’s approximately $0.22 / mile (figured at $4.50 / gallon for a 20 MPG car)

The Nissan supporting infrastructure — for example, their Leaf website and accompanying iPhone app — never seem to work. The car allegedly communicates quite a bit with the central station, but we’ve never been able to see the resulting information. The car itself is a gem, though.

When installed, the charger looks like this:

The Blink Charger has an embedded web server that can display the status of your month’s charges (that’s part of the reason for the subsidy — as I understand it they send anonymized information about your charging habits so that the government can get an understanding of electric vehicle usage).

It charges about 4KWh / hour, so to fill the car from empty would take about 6 hours. The Leaf comes with an emergency 120V charger, but that takes much longer.

Unfortunately, there was kind of a shock the first month:

Suddenly, our electricity bill was out of sight.

I really wanted to keep track of the charging schedule on the Leaf, to make sure that was the explanation for the entire leap. I didn’t see anything on the Blink website, and was afraid that I’d have to end up reading the web page from the Blink charger and parsing it, but it turns out that it was really cleanly implemented; you can get relevant information in JSON. For example, doing a:

curl http://your.blink.ip/cgi-bin/runcommand?coreMethod=chargestatus

will return something like

{
"returnCode": 0,
"chargeStatus": "CHARGESTATUS_UNPLUGGED"
}

The other legal charging status codes are CHARGESTATUS_CHARGING and CHARGESTATUS_STOPPED (i.e., plugged in but not charging anymore).

Also, you can get the last charge details by doing:

http://your.blink.ip/cgi-bin/runcommand?coreMethod=lastchargedetail

returning something like

{
"returnCode": 0,
"offPeakPercentage": 100,
"estimatedCost": "$0.80",
"startTime": "8:38 pm",
"startDate": "Jan 12, 2013",
"endTime": "1:10 pm",
"endDate": "Jan 13, 2013",
"elapsedTime": 5395
}

So, armed with a cron that would just query the Blink charger every 15 minutes, I was able to build a list of charging times for the car. PG&E in our area also allows you to download hourly usage, or for those with Smart Meters, one of the status indicators shows the current power draw for the entire house:

Here is it showing a current draw of 0.484 KWh.

After a few linear regressions, I found out that it was mostly the Leaf, but it was also the furnace. Given an unseasonably cold winter here in California, the furnace had been on substantially more than last year — and, the furnace draws about 0.6 KWh / hour to warm the house. (This is perhaps obvious — I mean, just because it’s a gas furnace doesn’t mean the entire thing is powered by gas — but it wasn’t obvious to me until I thought about it).

Upgrading to Jelly Bean

This was not obvious to me. (This will not be necessary to anyone getting a Nexus now, since Jelly Bean is preinstalled, but in case anyone is in my situation). After much Googling, I did:

• Install the Android SDK
• Used the WugFresh Galaxy Nexus Toolkit to unlock the phone.
• Installed Rom Manager from the Android store and had it install Clockwork. (Note that the WugFresh toolkit will also install ClockwordMod if you so desire).
• Got the Koush repack (referenced here) of Jelly Bean. The reason I went with this pack was because it’s odexed, which means that OTA updates will still work. Put the file anywhere.
• Boot into ClockworkMod by holding down power + volume buttons, pick the file from the previous step, and Jelly Bean is installed.

Moving from iPhone to Android

Remember to turn off iMessage in your settings before removing your card.

The iPhone 4 uses a micro-SIM, but it turns out that you don’t actually need an adapter to use the micro-SIM in the Galaxy Nexus — it fits snug enough that there is no problem. It took me a couple of tries to get it to work, though — the key is to line up the contacts just right. Specifically, the card does not go all the way to the left or right (or back) — it sits in the middle. A picture makes it more obvious:

I used My Contacts Backup to export all my contacts to a .VCF file, and then just imported it on the Android side.

I wanted visual voicemail (I’m on AT&T). The AT&T voicemail app got sketchy reviews and wanted to take over the messaging app, as well; I ended up using Google Voice for my voicemail messages.

Some people said that their connection was spotty and that they needed to change their APN settings, but I didn’t have any problems. Speed appears to be about the same as iPhone, although reception seems a little more iffy.

Hardware

The Galaxy Nexus is a nicely built phone, barring a slightly chintzy back panel — but, hey, you get a replaceable battery out of it. The only hardware buttons are the power button on the right side and a single volume rocker on the left; the Android buttons are displayed as soft buttons.

The Galaxy Nexus is a bit larger (apologies for the dark photo):

… but the larger size didn’t bother me, because I usually just stick the phone in my pocket anyways, and they both fit pretty handily. The larger screen is kind of nice, although it’s not as bright as the iPhone, nor is it as white. It may not be clear in the photo, but everything seems to have a distinct yellow cast on the Galaxy Nexus — whites on the iPhone are much whiter. While the Galaxy Nexus has a pretty high density screen, the Retina is still markedly better; small text on websites is easier to read, even on the smaller screen.

Battery life seems pretty comparable to iPhone, although the iPhone seems a little bit better (but, then, I had over a period of time eventually turned off everything I could on the iPhone, and haven’t yet done that process with the Galaxy Nexus). The phone itself seems faster than the iPhone 4, though; tasks such as bringing up a web page or loading the camera app are very quick.

Android vs. iOS

More notes on this after I’ve played with the phone for a week or so, but: Jelly Bean is a perfectly respectable operating system. It just lacks polish. Using Android is like having a beautiful house, but every room has a slow leak.

Some examples: go to marketplace, do a search, drill down on an object. Now, go back, but you’re at the beginning of the list again — you have to scroll back down. Or, in mail, almost any HTML formatted mail you open up is too wide, so you have to scroll the e-mail around to read it; you can’t zoom in and out in the Gmail app. The Google Books app turns pages by bending them in the middle, instead of turning the edge of the page first. Why do I need to use two e-mail apps (Gmail and Mail) for my mail, when it’s the same functionality? All minor stuff, admittedly, and might not even bother the majority of people that use the phone, but it bugs me a bit.

Some stuff is substantially better, though. The notification system in Jelly Bean is great, and kicks iOS’s notification system into a cocked hat. The flexibility of screen arrangement, and the ability to drop in arbitrary widgets and whatnot, is also a nice touch. Another great feature are the battery and data screens, which give you an app-by-app tally of what apps used the most battery / bandwidth.

The built-in turn by turn direction is really nice and better than my car’s built-in GPS, although a little clunky to find in the interface.

Google Voice, and the new info card system, seems completely useless. Simple requests like “Call ” or “Directions to home” aren’t understood. In fact, I can only get it do web searches, and the proactive card display that is supposed to be so amazing hasn’t worked at all for me. Mind you, the speech recognition part of it seems very decent, and it usually gets what I am trying to say, it’s just afterwards that it decides that “Call Carrie” means that I want to call Sears.

The Android marketplace has everything that I needed, although both the iOS store and the Android marketplace suffer from the problem of too many apps and not enough ways of filtering / sorting through them. (Example: searching for battery widget gave me 9500 results in the Android marketplace: kind of a pain to scroll through, and there doesn’t seem to be any way to sort).

More in a bit, after I’ve used the phone for a while.

Having the notification light is handy — I wish the iPhone had something similar.

Had to call Google Support for an issue with Google Play (I tried to order a rental, and if you order a rental on a Google account that doesn’t have Youtube available at the time, then the movie will not show up when you activate the Youtube account). I got a call back in 30 seconds, and they were helpful, and took care of my problem immediately! Amazing.

I understand that the screen is larger and will therefore consume more battery, but battery life is an issue. The phone seems to drain really quickly, even with everything off. I might have been spoiled by my iPhone 4, but I could go days between recharging — with the Galaxy Nexus, I always worried about getting through a day. I’m trying some of the battery saver apps to see if this gets better.

The Google Navigation is neat, but the voice overs are seriously bugged; all it does is repeat “in two miles” over and over again. Update: fixed this by clearing the data out for the Navigation app.

Well, the phone is growing on me. It’s missing polish, but it offers a substantial amount of customizability that is kind of nice. Tasker for Android is astounding — it allows you to define actions that are kicked off by arbitrary events. As an example, I set up my phone to:

• turn off Bluetooth and WiFi from 3:00am – 9:00am
• automatically bring up Stitcher (radio/podcasts) when I get into my car
• automatically read SMS messages received while I’m driving
• … so on and so forth

Another program that is amazingly handy is Mr. Number, which is basically a spam filter for phone calls. I used to get an inordinate amount of calls for, oh, termite inspections and cruise vacations and everything else under the sun, but Mr. Number filters them out without problems. I don’t see why this program is free — what’s their business model? — but it’s a must-have.

After installing Juice Defender and using aforementioned Tasker to turn off stuff at appropriate times, battery life is better — it uses about 1/2 % per hour in standby, and I can easily get through the day with over half my battery left, which is good enough.

Switched back to an iPhone (specifically, the new iPhone 5).

I will miss Tasker a bit, and Mr. Number quite a bit. In return I get a bluetooth stack that mostly works, and Siri on the iPhone 5 is everything that Google Voice wants to be, and I get iMessage with all my friends. Plus, battery life is substantially better.

All in all, it was worth using an Android phone for a while — and I could see using it again in the future, but for now I’ll stick with the iPhone ecosystem.

• Flycut (General Arcade) — stores your last n clipboards and allows you to easily choose one to paste. Saves a surprising amount of effort (for example, copying username / password? No more switching back and forth — just copy twice, switch to the terminal application, and paste twice).

Note: Emacs 24 appears to have a bug in their event handling that prevents Flycut from working properly. If you want to patch your version of Flycut to work properly with Emacs, you can see hacky fix here.

• Window Magnet (CrowdCafe) — makes it easy to snap windows to half or quarter screens. I bound the Ctrl-Alt-arrow keys to snap the window to half the screen.
• Iterm2 (George Nachman) — really nice terminal emulator with many features that I don’t use. Also has a fair number of profiles based on popular color schemes (i.e., Zenburn, Solarized, whatnot).
• Homebrew (also called Brew) — amazing replacement for MacPorts and Fink. Constantly being updated with new recipes for the latest stuff, and it’s easy to create your own recipes as well.
• Witch (Many Tricks) — nice window switcher that allows you to alt-tab to the right window, not just the right application. Lots of configuration options.
• Alfred (Running With Crayons, Ltd.) — spiritual successor to Quicksilver. Allows quick access to files and programs on your Mac. I use it to really reduce the size of my Dock Bar — anything I want to run is just a few keystrokes away.

I noticed that Fry’s had a sale on the Zotac AD02, so I picked one up fairly cheaply, and have set it up. The AD02 is a small little thing, with tons of connectors on the back (USB 2.0 + USB 3.0, eSata, HDMI, DVI). It’s fairly quiet — it does have a fan but the fan is very hard to hear over standard ambient noise — although it has to be said that they have a circular green LED on the top of the box that is bright enough for passing UFOs to notice and use as a landing spot.

One thing to note is that the Zotac AD02 eSata interface does have a port multiplier (PMP). With Ubuntu 11.10 I was able to connect to a Mediastation ProBox with multiple drives and access all of them without any problems.

BIOS Settings

I found it almost impossible to get into the BIOS. You’d think this would be easy. The trick is this: the Zotac AD02 defaults to a) a one-second window for going into BIOS setup, and b) numlock on. So, as soon as the box beeps, you need to hit Numlock and then delete. The Numlock is key, because if you don’t hit that then the delete key won’t register. When you’re in setup for the first time, set the boot delay to a bit higher and turn off Numlock.

Why? When you’re selecting a boot drive, Numlock submits the current selection. So, you see the list of drives that you want to boot, decide you want to move down 2, so you hit Numlock and… the machine starts booting.

Ubuntu 11.10

Either Zotac or Ubuntu doesn’t appear to boot from the USB 3.0 adapters — so, if you have a USB drive and you want to boot from it, you have to use one of the other 3 USB slots.

Ubuntu 11.10 32-bit installed without problems, but I noticed after installing that the drive was constantly flashing. Using iotop and echo 1 > /proc/sys/vm/block_dump to see what was going on; it revealed that that the jbd2 process (used by the ext4 journals) was constantly writing, which I eventually traced back to the lightdm greeter, which was spewing debugging messages into /var/log/x-0-greeter.log, of the form:

[+35196.63s] DEBUG: user-list.vala:1204: Drew 1 frames in 0.1 seconds
(9.932854 fps)

There’s a bug in Launchpad for this already, and it’s apparently been fixed recently. My incredibly hacky fix was to link that file to /dev/null, which was good enough for now.

sudo ln -s -f /dev/null /var/log/lightdm/x-0-greeter.log

For good measure I ended up putting a bunch of directories in tmpfs anyway. Ubuntu 11.10 has /var/run and /var/lock in tmpfs already, and I added a few others:

tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0 
tmpfs /var/log/apt tmpfs defaults,noatime 0 0

Note that putting /var/log into tmpfs is not generally recommended.

VNC

I usually log into the box via SSH but occasionally it is handy to be able to get to the GUI. I used the example described in this article to set up Upstart to automatically kick off X11VNC. The advantage of this method is that it’s started early enough that even the login screen is controllable through VLC.

XBMC

Putting in XBMC was straightforward, but you want it to use the graphics hardware on the E-350 chipset. I just used the instructions here and it went off without a hitch.

Well, the video was straightforward. I went to audio settings, set up everything for HDMI sound, and… no sound. And then I spent three days of little five minute sessions trying to get the sound to work.

I did not get it to work perfectly — but eventually ended up putting the following into /etc/asound.conf:

pcm.!default {
  type plug
  slave {
    pcm "hw:0,3"
    rate 48000
  }
}

… and then setting audio in XBMC to plughw:0,3, I was able to get videos and music working (but no menu sounds). For something that’s only occasionally a XBMC server, this seemed good enough.

Anyway, I can play a video and the CPU is only running at about 20%, so there are pretty of cycles left to run a torrent or index my mail or whatever else.

Air Video

I also set up an Air Video server according to the instructions here, as there’s a PPA available that makes it incredibly easy to install. Unfortunately, both CPUs are almost completely pegged when streaming with Air Video, but it is good enough to stream the occasional show.

Step 1: Install msmtp (or any other program)

On OS X you can do this with “brew install msmtp”. (You can use sendmail or whatever, but msmtp was easy to set up.).

Step 2: Set up msmtp

Set up the appropriate configuration file. I ran into a problem wherein the program would just hang; turning on debug showed that it hung after “Reading recipients from command line.” This turned out to be an incorrect SSL configuration — I needed tls_starttls to be off.

My particular .msmtprc configuration:

defaults
tls on
tls_starttls off
tls_certcheck off
 
account Work
host mail.host.com
domain domain.com
auth on
port 465
protocol smtp
from from-name
user user-name
password password

Step 3: install sendmail replacement

For me, this was two things: a small shell script and a Python program that will convert any plaintext e-mail into a plaintext + HTML e-mail.

Shell script mark_send:

#!/bin/sh
/Users/tim/mutt/mark_and_send.py "/usr/local/bin/pandoc -s" | /usr/local/bin/msmtp -a Work $@

Python script mark_and_send.py:

#!/usr/bin/python
import os
import sys
import subprocess
import email
import email.parser
import tempfile
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
 
if len(sys.argv) == 1:
    print "usage: mark_and_send.py markdown_program [markdown_flag]"
    print "where:"
    print "    markdown_program is the name of the program to translate markdown."
    print "    markdown_flag is an optional indicator in the first line that determines whether to run the markdown program"
    sys.exit(0)
 
markdown_program = sys.argv[1]
 
p = email.parser.Parser()
m = p.parsestr(sys.stdin.read())
 
# is this a single header?
do_markdown = False
if not m.is_multipart() and m.get_content_maintype() == 'text' and m.get_content_subtype() == 'plain':
    do_markdown = True
    payload = m.get_payload()
    # optionally check the markdown flag
    if len(sys.argv) == 3:
        lines = payload.split("\n")
        do_markdown = lines[0] == sys.argv[2]
        if do_markdown:
            payload = "\n".join(lines[1:])
 
if do_markdown:
    tf = tempfile.NamedTemporaryFile(delete=False)
    tf.write(payload)
    tf.close()
 
    (markdown, error) = subprocess.Popen(markdown_program.split() + [tf.name], stdout=subprocess.PIPE).communicate()
 
    # create a new message with the exact same headers
    new_message = MIMEMultipart('alternative')
    for (k,v) in m._headers:
        new_message[k] = v
 
    text_plain = MIMEText(payload, 'plain')
    new_message.attach(text_plain)
 
    text_html = MIMEText(markdown, 'html')
    new_message.attach(text_html)
 
    print new_message.as_string()
    os.unlink(tf.name)
else:
    print m.as_string()

The mark_and_send.py program takes one required argument (the name of the program to convert from markdown to HTML, possibly quoted) an an optional indicator of markdown. If the first line of the message is the indicator, then it is stripped and the rest of the message is converted to HTML.

Step 3: set up mutt

You need to tell mutt to use your shell script instead of sendmail. I have it inside a folder hook:

folder-hook +work/.* 'set sendmail="/bin/sh /Users/tim/mutt/mark_send"'

Now, whenever you send a plain-text email, mutt will automatically run markdown on it and attach it as the HTML view of the message.

Results

• Markdown is not ideal for e-mail, at least, not without some work. For example: more e-mails that are quoted by default will not work properly in markdown, because there are not blank lines in the message to set off the blockquote. This could possibly be improved by post-processing the file that Mutt generates on a reply, but I don’t think there’s a hook for that.
• Code coloring is nice
• Math formulas, if you need them

Example is show below, as rendered by Zimbra. It isn’t perfect. Google will unfortunately strip the colors from the code coloring, and Zimbra loses leading indents, and both readers will require the user to load images. Nonetheless, in a pinch…

• usb_buffer_free and usb_buffer_alloc had changed in the most recent kernel
• needed to add a device code for my adapter
• did not need to update the firmware or the USB list

• download files from the blog
• show differences
• copy changes wholesale from the blog to local, or vice versa
• add new posts

It’s all on Github.

Setting Up

Assuming that your blog is set up at http://www.yourblog.com (you can create an account at wordpress.com to test this out), all you’ll need to do is:

-- make a directory for the blog
mkdir blog
chdir blog

-- download everything
python wp.py --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php init
(wait a bit)

-- now set up so we don't have to specify --user, --password, and --url every time (optional)
python wp.py --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php defaults
-- you can skip specifying the password, and it'll prompt you when you run it

The files are downloaded in the appropriate YYYY/MM directories, with the draft directory being used for all of your unpublished drafts.

All the drafts are stored in plain text, but you’ll see some lines starting with periods — these are various WordPress variables that are associated with the file. You can change them, as well; for example, to change the title of the post, just change the line that begins with “.title”.

Seeing What’s Different

You can use the status command to see differences between the local file system and your blog.

python wp.py status

Note that only the most recent files are checked. If you want to really check every single file for changes, do:

python wp.py status all

You can also use the –diff command line option to see the differences between local and server:

python wp.py --diff status all

Updating From The Blog

If you’ve made changes through the web interface and you’d like to bring them down, you don’t have to download everything again, but can instead just update.

python wp.py pull

Again, only the most recent changes are brought down. If you want to check every post on the blog, do:

python wp.py pull all

Posting To The Blog

If you’ve made changes to files and you’d like to post them back, do:

python wp.py push

To push everything (and not just the most recent files), do:

python wp.py push all

Note that push only changes those files that exist in both spots. If you’re adding a new post, use the “post” command.

Posting/Editing

If you’d like to add a new post, put it in the drafts folder, and then do:

python wp.py post drafts/filename

Note that add can actually take existing posts, as well — it just forces an update of that one file, rather than running through all changes like push.

Publishing

To publish a file, just change the .post_status field from ‘draft’ to ‘published’. Note that doing this will cause a copy to move from the drafts folder to the appropriate year/month.

Gotchas

There are some gotchas due to the fact that the filename can change on you. There are cases where the filename that will be brought down is different then the one that you send up:

• You post a file without a .title or .wp_slug line
• You post a file with a different file name than the slug that is generated (i.e., “my-first-draft” when the title is actually “my final draft”)

The Google code provides a command line program that can generate secret keys as well as a PAM module, but it turns out to be very little code to authenticate a TOTP, thereby providing two-factor authentication to your website very easily.

To give the user the key, you’ll need to generate a cryptographically-secure 10 byte random key, presented to the user as a base32 16-character string. They can either enter this string directly, or you can use Google charts to provide a barcode that they can scan into the Google Authenticator application:

def get_barcode_image(username, domain, secretkey):
    url = "https://www.google.com/chart"
    url += "?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/"
    url += username + "@" + domain + "%3Fsecret%3D" + secretkey
    return url

For an example of what a code looks like, click here, or, look below:

After the user has a secret key from you and has entered it into Google Authenticator either by typing it in directly or scanning in the barcode, you have to be able to verify the key during login (for example). The code to authenticate is only a few lines in Python:

import time
import struct
import hmac
import hashlib
import base64
 
def authenticate(secretkey, code_attempt):
    tm = int(time.time() / 30)
 
    secretkey = base64.b32decode(secretkey)
 
    # try 30 seconds behind and ahead as well
    for ix in [-1, 0, 1]:
        # convert timestamp to raw bytes
        b = struct.pack(">q", tm + ix)
 
        # generate HMAC-SHA1 from timestamp based on secret key
        hm = hmac.HMAC(secretkey, b, hashlib.sha1).digest()
 
        # extract 4 bytes from digest based on LSB
        offset = ord(hm[-1]) & 0x0F
        truncatedHash = hm[offset:offset+4]
 
        # get the code from it
        code = struct.unpack(">L", truncatedHash)[0]
        code &= 0x7FFFFFFF;
        code %= 1000000;
 
        if ("%06d" % code) == str(code_attempt):
            return True
 
    return False

I wanted an easy way of running Clojure programs under Hadoop, and ended up writing a silly, simple little shim that would simply take a Clojure file with a mapper and reducer function. It means that there is no .JAR building and no AOT compiling — just write your mapper and reducer function, bundle up the file, and go. Note that the JAR is built for Hadoop 0.20 and later.

Getting The Jar

The quickest way of downloading the .JAR is to just download it.

Building The Jar

Build the shim:

javac -cp /usr/lib/hadoop/hadoop-core.jar:/usr/lib/hadoop/lib/*:lib/* -d classes Shim.java

Create a lib directory and add the clojure classes:

mkdir lib
cp /from/wherever/clojure-1.2.0.jar lib
cp /from/wherever/clojure-contrib-1.2.0.jar lib

Bundle it together into one .JAR file:

jar -cvf shim.jar -C classes/ . lib/*

[~/github/hadoop-shim] $ jar tf shim.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/brool/
com/brool/Shim.class
com/brool/Shim$Reduce.class
com/brool/Shim$Map.class
lib/clojure-1.2.0.jar
lib/clojure-contrib-1.2.0.jar

Using The Jar

To run the example wordcount:

hadoop jar shim.jar com.brool.Shim -files wordcount.clj input-file output-file

The output from the run will be in output-file/part-r-00000

More Details

The Clojure file that you provide should be in the user namespace, and must provide two functions named mapper and reducer.

Mapper

The mapper function takes a string representing one line in the input file and returns a list of [ key value ] pairs. A given input line can generate any number of map lines.

Reducer

The reducer is given a key and all values that were associated with that key. The reducer’s function is to consolidate all of that into one output line.

Example

Since word count is the canonical example, let’s do that. The mapper for word count is:

(defn mapper [v]
  (let [words (enumeration-seq (StringTokenizer. v))]
    (map #(do [ % "1" ] ) words)
))

As an example, for an input line of “This is a test, is it not?”, the following map will be generated:

(["This" "1"] ["is" "1"] ["a" "1"] ["test," "1"] ["is" "1"] ["it" "1"] ["not?" "1"])

Before being given the reducer, the Hadoop system will group all like keys together, resulting in:

"This" => [ "1" ]
"is"     => [ "1" "1" ]

So on and so forth. The reducer is given the key and the list of all values that had that key, and then emits the final result — for a word count, the correct code would be:

(defn reducer [k v]
  [ k (reduce + (map #(Integer/parseInt %) v)) ])

This simply adds up the counts.

Testing

In wordcount.clj there are two handy functions that allow for debugging of the mapper and reducer before submitting it to Hadoop.

Given a local file, the test-mapper will load the file and run it through the mapper; if your file is large you may just want to (take 20 (test-mapper “/my/filename”)).

The test-reducer function will load the file and run it through both the mapper and the reducer. Taking the example sentence above:

user> (test-reducer "/tmp/one-sentence")
(["This" 1] ["a" 1] ["is" 2] ["it" 1] ["not?" 1] ["test," 1])

Example #2

As another example: let’s say that you have a collection of log entries, and would like to record the first and last log entry for every user. Assume that the files are in a CSV format, with the fields being in the order of timehit, userid. Example:

2010-10-04 13:04:22,112334
2010-10-04 10:04:22,182994
2010-10-04 10:05:18,182994
2010-10-04 10:07:19,182994
2010-10-04 13:28:41,112334
2010-10-04 10:09:22,182994
2010-10-04 13:56:22,112334
2010-10-04 11:30:01,182994

The mapper for this:

(defn mapper [v]
    (let [[timehit userid] (.split v ",")]
        [ [ userid timehit ] ]
))

The reducer:

(defn reducer [k v]
(let [s (sort v)]
[k (str (first s) "," (last s))]))

The source is all on Github.

Beautiful Soup is a really nice Python library for extracting content from possibly-sloppy HTML, and I wanted some reasonably close Clojure equivalent. Unfortunately, the standard classes don’t work well with malformed HTML; as an example:

    => (require '(clojure [xml :as xml]))
    => (xml/parse "http://www.google.com")
    org.xml.sax.SAXParseException: The markup in the document preceding the root element must be well-formed. (NO_SOURCE_FILE:0)

Fortunately, there is already a TagSoup library that can parse non-perfect HTML, and it is very easy to integrate TagSoup into xml/parse. This module hardly does anything; it simply adds a few helper routines and brings the most-used calls into one amazingly bad namespace name.

Examples

Building your soup:

    (use beaujiful-soup.core)
 
    ; build soup from URL
    (def t (build-soup "http://www.google.com"))
 
    ; build soup from (deliberately malformed) string
    (def t2 (build-string-soup "<html><body><ul><li>One<li>Two</ul></body></html>"))

Extracting information is done with the xml-> call. Oftentimes the last thing you do will be a node or text or (attr :attribute) call, in order to convert the results into a more workable type:

    ; you can "walk" down the tree with successive tag names.  For
    ; example, get every list item inside the unordered list
    ; immediately inside the body.
    (xml-> t2 :body :ul :li node)
    ; => ({:tag :li, :attrs nil, :content ["One"]} {:tag :li, :attrs nil, :content ["Two"]})
 
    ; get the text for the list items
    (xml-> t2 :body :ul :li text)
    ; => ("One" "Two")
 
    ; Get textareas immediately inside the body.
    (xml-> t :body :textarea node)
    ; => ({:tag :textarea, :attrs {:id "csi", :style "display:none"}, :content nil})
 
    ; use descendants to iterate through all nodes, not just the immediate children.
    ; Get the text from all <a> tags anywhere in the body.
    (xml-> t descendants :a text)
    ; => ("Images" "Videos" "Maps" ...)
 
    ;  Get the href attribute from all tags
    (xml-> t descendants :a (attr :href))
    ; => ("http://www.google.com/imghp?hl=en&tab=wi" ... )

Use the (attr=) predicate to match an attribute value:

    ; find invisible stuff
    (xml-> t2 descendants (attr= :style "display:none") tag)
    ; => (:textarea :iframe)

Strings match the text inside nodes:

    ; find the link for the <a> that has "Videos" for content
    (xml-> t descendants :a "Videos" (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Arbitrary predicates can be used as well. They will take a loc (location), and are usually converted to a node before being used:

    ; find any :p or :div
    (defn p-or-div [loc] (contains? #{:p :div} (:tag (node loc))))
    (xml-> t descendants p-or-div tag)
    ; => (:div :div :div :div :div :div :div :div :div :div :div :p :div :div)
 
    ; find the link for <a> that has case-insensitive "Videos" for content
    (require 'clojure.string)
    (defn f [loc] 
      (let [n (node loc)]
       (and (= (:tag n) :a) (= (clojure.string/upper-case (first (:content n))) "VIDEOS"))))
    (xml-> t descendants f (attr :href))
    ; => ("http://video.google.com/?hl=en&tab=wv")

Fundamentally, the xml-> call returns a list of locations, and you can apply arbitrary transforms as necessary. For example, let’s say that you want to build a map of text => hrefs for all of the links:

    (defn loc-to-pair [loc]
        [ (attr loc :href), (text loc) ])
    (apply hash-map (xml-> t descendants :a loc-to-pair))
    ; => {"/services/" "Business Solutions",  ... }

Having a vector in the chain applies all the predicates within the vector, and filters out anything that doesn’t match. It acts a little like a lookahead in a regex. For example:

    ; Find the IDs of all divs that contain an href immediately within them
    (xml-> t descendants :div [ :a ] (attr :id))
    ; => ("fll")
 
    ; Find the IDs of all divs that contains an href anywhere within them
    (xml-> t descendants :div [ descendants :a ] (attr :id))
    ; => ("ghead" "gbar" "guser" "fll")

Source

• Display is ridiculously nice
• It’s fast — subjectively, as fast as the iPad — and the extra CPU power really makes the web browsing much better.
• The bumper case is not that great — it prevents some headphones from being used (because the distance to the headphone jack is a bit longer) and also prevents you from using the 10,000 iPod USB adapters that you have lying around the house, because the slot in the case for the iPod adapter is just small enough that older adapters don’t work. As one of my friends said, “Get the Dremel!”
• I took the case off and tried to reproduce the reception problems that people had been talking about, but was unable to get it to happen.

The ominous thing was that the iPhone 4G suffered from the same wireless problems as the iPad, manifesting as a couple of symptoms: a) unable to connect unless I was really close to the router, b) and really low signal strength.

Wireless

I had been living with the wireless problems for a while, although they were extremely frustrating. Finally, a lucky break meant that I was able figure out the underlying cause (although to my chagrin I discovered later that other groups had identified this a long time ago). My iPhone 4G wouldn’t get onto the net, and even worse, every time I tried to get onto the net, it would kick my laptop off for a minute or two.

At that point, it became blindingly obvious — the phone was trying to use an expired lease, and as a result was conflicting with the laptop. Thirty seconds later I had set up static IPs for the iPad and iPhone 4G, and all my problems were solved. I could use the devices all over the house, there were no mysterious drops, signal strength was fine, and as a bonus, the mysterious wireless network failures that my laptop was having were solved.

Lessons Learned

Well, I really should have done my research — I could have found the answer to this much earlier.

Why did I think it had something to do with signal strength? I suspect that the 3.2/4.0 iOS somehow bases the signal strength indicator on DHCP status — where I was getting one bar before is now three. The alternate explanation is that I tended to use the devices in the more remote places at night, which happened to coincide with the DHCP conflict.

(defproject mydirectory "1.0.0-SNAPSHOT" :description "FIXME: write" 
  :dependencies [[org.clojure/clojure "1.1.0-alpha-SNAPSHOT"] 
    [org.clojure/clojure-contrib "1.0-SNAPSHOT"]
    [mysql/mysql-connector-java "5.1.6"]
    [incanter/incanter "1.0-master-SNAPSHOT"]])

(that is, add the mysql and incanter dependencies).

Download all the dependencies with lein deps

Start up a REPL with everything in the classpath by just using lein repl

Wow, that’s kind of nice.

Unfortunately, despite the fact that there were some good guides on how to do it, it turned out there were a few gotchas, and it took significantly longer than 10 minutes. So, in the hopes that I can save someone else the time:

Use this guide, but:

• in Windows 7, you’ll want to use the “Network and Sharing Center” (right click on your network icon) to create a new “computer-to-computer” network. This is the one the iPhone will connect to.
• When you configure Putty, you’ll want to make sure you go to the SSH/Tunnels page and a) add a dynamic tunnel to a local port (assume 8080) — you won’t need to specify a destination and can leave that blank, and b) go to the “Connection” page and specify a keepalive of 5 or so. This keeps the SSH connection between the computer and the iPhone from dying every 30 seconds.
• You cannot use Chrome right now, because it does not properly forward DNS requests to the proxy (it took me hours to figure this out). Use Firefox and specify a SOCKS v5 proxy at port 8080 (or whatever) for the proxy, but you’ll also need to make sure to go to the about:config page and change the network.proxy.socks_remote_dns to “true”.
• Doing things in a particular order makes this work better. If the 3G connection on the iPhone is not active, then there doesn’t seem to be a way to activate after it links to your ad hoc connection. So: turn off wireless, go to a web page (this activates the 3G), turn on wireless, connect to your computer. This way both the 3G and wireless connections are established. An easy way to see if the 3G is active is to run MobileTerminal (or just SSH into the iPhone) and run ipconfig; if the pdp_ip0 device is pointing to a real IP, then your 3G is active.

Man, netbooks have really come a long way. A quick run through:

Screen: 1366×768 vs. 1024×600 is almost twice as big, and makes a big difference. It’s glossy, which is kind of annoying, and I think the contrast is a little lower than on the 1000h, but nonetheless it’s really nice.

Keyboard: Full sized shift keys on the Acer! This far outweighs almost anything else. The smaller cursor control keys are kind of pain, but I’ll get used to them. Feedback is not as nice as the Asus but still pretty decent — I can type full speed on this keyboard.

Touchpad: Touchpad is good enough; at least there are two mouse buttons, which is one of the things I really wanted. It’s multitouch, but the drivers are a little choppy with the two finger scrolling — but I haven’t tried to fix it, either. The two finger to zoom in and out is surprisingly handy when browsing, to be honest, and I’m using it all the time.

Wireless: Wireless on my EEE 1000h was always a little glitchy at home (and would sometimes cause my router to reboot, for whatever reason), but was fine at work and at Starbucks. The wireless on the Acer 1410 works much better, and I haven’t had any problems.

Speaker: I frequently watch movies with the speakers on. The speakers seem to be just a touch softer than the EEE 1000h, but still pretty good. This tweaks thread has some tips on improving the volume of the speakers.

Processor: The new ULV chips are everything that Atom wanted to be. With the Acer I’ve been able to a) run Ubuntu under virtualbox, b) play full-screen 480p Hulu, and c) play full-screen Netflix, and it hasn’t stalled out on anything. Additionally, the 1000h would bog down when scrolling in Firefox (seriously), whereas this machine handles it fine.

I specifically looked for an Acer with the SU2300 (dual core) instead of the SU3500 (single core, slightly faster).

The machine only starts up the fan when doing flash video — fan noise is not too bad and significantly quieter than the EEE 1000h. The computer itself does not get very warm at all, which is nice.

VirtualBox was a must have, since I wanted to run Ubuntu but didn’t want to deal with the pain of partitioning and dealing with its peculiarities on a new machine. I had assumed that having the AMD/Intel virtualization mode was a big win, but it turns out that it really doesn’t help performance. Nonetheless, Virtualbox works fine. VT-x mode was originally turned forced off, and it was subsequently fixed in the BIOS; see this tweaks thread for tips on getting the new BIOS.

Battery Life: Hard to say, but looks like 5-7+ with this machine, whereas I was getting about 4 with the Asus 1000h.

Operating System: I run XP on most of my computers (and didn’t upgrade any to Vista), but the Acer came with Windows 7 Home Premium. It’s pretty and nice and performs well — I have no complaints.

What a difference a year makes. These new ULV chips are amazing.

I really enjoy Clojure. Everything seems so well thought out and well designed; in a lot of ways it reminds me of Python, which is three ways to ironic because back in the day I started using Python because it was very Lispy. The cycle of life turns even on computer languages.

But unfortunately, I get the sense that there’s probably only room for one JVM language to make it big. As much as I like Clojure, I think that Scala will probably be the one that succeeds. It’s got the static typing, it’s got the Java-like performance, and it’s got a syntax that’s close-ish to Java, so it fills the niche that Java fills so well: a programming language that is ideal for large groups of developers.

However, it’s early enough in the development of Clojure that there’s the sense that maybe individual efforts could make a difference. (As a counterexample, Haskell is too far along — it seems like everything has been done already, or anything that needs to be done requires a deep knowledge of category theory and the latest functional design papers.)

Assuming that I would want to help out Clojure, what could be done?

Donate? Actually, I did that. I figured that I got as many hours of enjoyment out of Clojure as I did out of a typical game, so I contributed the price of a game.

Libraries? Well, I’ve written one or two. Will probably write more.

If Clojure is going to succeed, it needs a niche, an area in which it performs so well that it blows everything else away. Concurrent programming is one possibility, but there is a lot of competition there.

I have a modest proposal:

Clojure should be more like PHP.

Yes, yes, I know that programming in PHP is socially reprehensible, just one step above people that shop without their shirt while in Walmart and car salesmen.

But what if we were to give Clojure the trappings that make starting developers try it? Metaphorically, it’s like Gulliver’s Travels; it looks like a humorous, light-hearted adventure story, but underneath it’s biting political satire and deep thoughts. We lure developers in with the possibility of an easy, light-hearted way to write web sites… and then once they’re in there, once they’ve progressed from manipulating web sites and start to realize the benefits of advanced languages, they can deal with a sane language, built on top of the JVM and with the full power of a Lisp.

So call this Gulliver.

Why Did PHP Succeed?

My impression, based on absolutely no citations or hard evidence, is that PHP became popular primarily because it is:

• Easy to start trying — start with an HTML page, and just add one statement to start playing around with PHP itself
• Easy to access data — typically, with MySQL
• Easy to deploy — just upload a file and go

So, continuing the gedankenexperiment: if we were designing something that would be very approachable and very easy to use, what would it look like?

Making Compojure More Approachable

Compojure is an excellent base to start with, but it strays away from templates, and instead tries to do everything with compojure.html, a library that allows you to form everything with vectors and lists:

(html [:p "2 + 2 = " (+ 2 2)])

Handy? Yes, if you’re a developer. But if a web designer is doing your HTML, or if you’re just starting out programming and you already know HTML, though, it seems an additional barrier to entry. So what if we turn it inside out and do it like PHP does it?

<p>2 + 2 = <?= (+ 2 2) ?>

Trying something more complicated, in compojure.html this:

(html [:ul (map (fn [x] [:li x]) (range 10))])

<ul>
  <? (dotimes [x 10] ?>
  <li><?= x ?>
  <? ) ?>
</ul>

A little bit longer, but treating it as HTML makes it conceptually easier for beginning programmers to understand, especially if you want to do something like changing the id or class with every row so your CSS can make it all pretty:

<ul>
  <? (dotimes [x 10] ?>
  <li id="list-<?= x ?>" class="<?= (if (even? x) "even" "odd") ?>"><?= x ?></li>
  <? ) ?>
</ul>

Doing this is compojure.html is trickier:

(html 
  [:ul 
    (map (fn [x] [:li {:id (str "list-" x) :class (if (even? x) "even" "odd")} x]) 
    (range 10))])

So, let’s lure people in with easy to use, and they can pick up vectors and S-exprs and notational convenience later. Converting a template file into the basic code is easy:

(defn- transform-string
  "Internal.  Transform a string into the appropriate echo statement.
An empty string translates into a single space."  
  [s]
  (if (> (.length s) 0)
    (format "(gulliver/echo %s)" (pr-str s))
  " "))
 
(defn- transform-code
  "Internal.  Transform code so that it can run as a render routine."
  [s]
  (if (= (first s) \=)
    (format "(gulliver/echo %s)" (.substring s 1))
    s))
 
(defn- convert* 
  "Internal.  Given a string that represents a Clojure template,
produce a flat list of the statements necessary to produce that template."
  [s]
  (let [m (re-matcher #"(?s)(.*?)<\?(.+?)\?>|(.+?)$" s)]
    (loop [next (re-find m) buffer ["'("]]
      (cond
       (not (nil? (nth next 3))) 
           (eval (read-string (apply str (conj buffer (transform-string (nth next 3)) ")" ))))
       :otherwise (recur 
                   (re-find m) 
                   (conj buffer 
                         (transform-string (nth next 1)) 
                         (transform-code (nth next 2))))
))))

Resulting in…

==> (convert* "<ul><? (dotimes [x 10] ?><li><?= x ?></li><? ) ?></ul>")
 
((gulliver/echo "<ul>") (dotimes [x 10] (gulliver/echo "<li>") (gulliver/echo x) (gulliver/echo "</li>")) (gulliver/echo "</ul>"))

Once you have the basic code generation, it’s easy enough to hook it up to a default servlet that automatically invokes it when necessary. There’s a fair bit of handwaving going on here — regexes are questionable for this, there are some interesting issues with namespaces, and it would require some method of resolving dependencies between files, but heck, it’s only 100 lines of code; it was really quick to put together and gives us a sense of whether this can work out.

So, with the source on github, your basic server becomes…

(use 'gulliver)
(defserver web-server
  {:port 8080}
  "/*"  (servlet gulliver/template-servlet))

Easy! As an test, an example file:

<?
(import 'java.util.Date)
?>
 
<html>
  <body>
    <h1>Delivered via Gulliver</h1>
    <p>Expressions at work: <?= (+ 2 2) ?></p>
    <p>Looping:</p>
    <ul>
    <? (dotimes [x 10] ?>
      <li><?= x ?></li>
    <? ) ?>
    </ul>
 
    <p>Here's a date:  <?= (Date.) ?></p>
  </body>
</html>

… does the right thing. How about a time test?

<html>
    <body>
    <h1>Starting time test...</h1>
    <? (let [ start (System/currentTimeMillis) ] ?>
    <p>Adding 1..100000:  <?= (reduce + 0 (range 1 1000001)) ?></p>
    <p>(took <?= (- (System/currentTimeMillis) start) ?> ms)</p>
    <? ) ?> 
    </body>
</html>

… runs in about 50ms. PHP takes about 130ms on the same sequence on my laptop, so, while there are tons of optimizations we could do (like not doing a stat on the file every time the page is rendered (!)), it nonetheless performs well enough that maybe this is a viable option. Finally, as a special bonus, a stupid eval trick that evaluates Clojure code:

<html>
    <body>
        <? (when ((request :params) :code) ?>
        <pre style="border: 1px dotted">
        <?= (str (eval (read-string ((request :params) :code)))) ?>
        </pre>
        <? ) ?>
        <form method="post">
        <textarea name="code" rows=10 cols=80></textarea><br/>
        <input type="submit" value="Evaluate" />
        </form>
    </body>
</html>

Take that, non-dynamic languages!

Github copy of the source

Next part: Schema-ing Against MySQL

public void OnIdleStateChange(AccSession arg0, int arg1) {
}
 
public void OnInstanceChange(AccSession arg0, AccInstance arg1, AccInstance arg2, AccInstanceProp arg3) {
}
 
public void OnLookupUsersResult(AccSession arg0, String[] arg1, int arg2, AccResult arg3, AccUser[] arg4) {
}
 
public void OnSearchDirectoryResult(AccSession arg0, int arg1, AccResult arg2, AccDirEntry arg3) {
}
 
// ... go on like this for pages

The Java code is here, if you’re interested in the entire set of calls. Now, I didn’t care about most of those events, but I had to override them, since they didn’t have a default implementation. What made this seem painful was that I was really only interested in two of the callbacks. So I started to record an Emacs macro to convert the Java code to the equivalent Clojure proxy statement, and then I realized that I didn’t have to — I was using a Lisp.

(defmacro auto-proxy [interfaces variables & args]
  (let [defined (set (map #(str (first %)) args))
        names (fn [i] (map #(.getName %) (.getMethods i)))
        all-names (into #{} (apply concat (map names (map resolve interfaces))))
        undefined (difference all-names defined) 
        auto-gen (map (fn [x] `(~(symbol x) [& ~'args])) undefined)]
    `(proxy ~interfaces ~variables ~@args ~@auto-gen)))

Auto-proxy works just like proxy, but it makes an empty implementation for any call that wasn’t defined. So, suddenly, what would have been a bunch of lines collapsed into just:

(defn create-aim-proxy []
  (auto-proxy [com.aol.acc.AccEvents] []
     (OnImReceived [session imSession participant im] 
        (handle-im session imSession participant im))
     (OnStateChange [arg0 arg1 arg2]
        (handle-state-change arg0 arg1 arg2))))

Macros ftw. The nice thing about Clojure/Lisp is that it makes coding up this kind of reusable framework stuff really easy.

Clojure code density seems to be pretty good. There are a fair number of convenient shortforms in the language; for example, associative datatypes all act as a function — so given a hash map you can reference it with (my-hashmap :key). The base language itself is probably about as expressive as Python (or a bit better), but you have the added advantage of being able to use macros as needed to really get the code density up.

Nonetheless, I really wanted something like Ocaml’s / Haskell’s pattern matching; it makes some code wonderfully concise.

Accordingly, I hacked something up, based on Clojure’s built-in destructuring. Some examples:

Literal values match against the same value, while _ matches against any non-nil value (and nil matches against any nil one). Additionally, :when clauses can be used for conditional checks.

; simple recursive evaluator
(defn arithmetic [lst]
  (match lst
    v  :when (number? v)  v
    [ _ "error" _]     "error"
    [ _ _ "error"]     "error"
    [ "add" a b ]      (+ (arithmetic a) (arithmetic b))
    [ "sub" a b ]      (- (arithmetic a) (arithmetic b))
    [ "mul" a b ]      (* (arithmetic a) (arithmetic b))
    [ "div" a b ]      (/ (arithmetic a) (arithmetic b))
    [ "squared" a ]    (arithmetic ["mul" (arithmetic a) (arithmetic a)])
    _                  "error" ))

Both collections and single values can be used:

;; return signum fr a number
(defn signum [x]
  (match x 
     0 0
     n :when (< n 0) -1
     _ 1))

The pattern matching is stricter than the typical destructure; whereas [ a b ] will destructure against a list of any number of elements, [ a b ] will pattern match only against a list of two elements.

(match x 
    []    "empty"
    [_]   "one element"
    [a a] "two identical elements"
    [_ _] "two elements"
    _     "three or more")

If the same variable occurs in multiple locations in the parameter list, it will be checked for equality. The & tail form can be used to specify the rest of the list.

;; count identical elements in the same location in two lists:
(defn count= [ lst1 lst2 ]
  (loop [ a lst1 b lst2 count 0 ]
    (match [a b]
      [[e & at] [e & bt]]  (recur at bt (inc count))
      [[_ & at] [_ & bt]]  (recur at bt count)
      _                    count)))

Note that this is slightly more flexible than Haskell / ML, in that a variable of the same name can be multiple places in the pattern.

Defining

You can use the defnp macro to define a function that is pattern matched; it defines a function that takes one argument and has an implicit match statement. For example, the signum function can be written:

    (defnp signum
       0 0
       n :when (< n 0) -1
       _ 1)

(Thanks to Tom Faulhaber for suggesting this)

Gotchas

The Clojure destructuring will cause an exception if you try to destructure a collection type with a value.

(let [[a b] 10] a)
java.lang.UnsupportedOperationException: nth not supported on this type: Integer (NO_SOURCE_FILE:0)

… so be sure to check such cases early in your match statement, if they are possible.

How It Works

The pattern matcher uses the built-in Clojure destructuring as the main mechanism, but adorns it so that the pattern can be verified. For example, the code:

(match x [a a] "two identical" )

(let [ [ a g0001 & g0002 ] x ] 
     (if (and (not (nil? a)) (= g0001 a) (nil? g0002)) "two identical" nil))

That is, the destructuring is done, but then the two variables are checked to make sure that they are equal, and the list is checked to make sure it is only two elements long.

Source

Nonetheless, I ran into a situation where it seemed to be 100x slower than Java because of an erroneous assumption on my part. After stripping out and profiling and removing everything extraneous, it finally came down to array access. Ignoring the uselessness of this snippet for a while, the issue is how to translate something like the following code:

... which runs in about 350ms on my Macbook. My pass at the Clojure equivalent, with the type hinting:

;; this is 100x slower than the equivalent in Java
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]
        (aset-int v ix (unchecked-add (int 1) (aget v ix)))
        (aset-int v ix (unchecked-subtract (int 1) (aget v ix)))))))

... which takes about 40692ms -- more than 100x slower. What the hell? After many hours of Googling, I finally discovered from the comments on this blog post that aset-int is slower than aset, which I don't recall seeing anywhere else. Does it really make a difference?

;; much faster
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]>
        (aset v ix (unchecked-add (aget v ix) (int 1)))
        (aset v ix (unchecked-subtract (aget v ix) (int 1)))))))

... runs in about 1000ms, or about 3x slower than the Java variant, and is definitely something that I can live with. I had been assuming that aset-int was of course faster because it was more specific, but in fact (aset v ix (int val)) is much faster than (aset-int v ix val).

At any rate, I made a small, thin layer around the Tokyo Cabinet API, and put it on Github. Another thin wrapper can be found at this blog.

Copy of the README is below (the ultimate in lazy!).

Introduction

This is a simple interface to the Tokyo Cabinet libraries. Tokyo Cabinet is a very cool, very high performing key-value store. This library supports table mode, which essentially means that arbitrary hashmaps can be stored in the cabinet.

Note that this is appropriate for local storage only — if you’re looking to share a Tokyo Cabinet to multiple computers, you actually want Tokyo Tyrant.

Basic Usage

The with-cabinet call creates/opens a cabinet and allows the use of the various access routines within the scope of the call. For example, here’s how to create a cabinet with three entries.

(ns user (:use tokyo-cabinet))  ;; bring into our namespace
 
(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREAT) } 
    (doseq [[name val] [["1" "one"] ["2" "two"] ["3" "three"]]]
        (put-value name val)))

This creates a Tokyo Cabinet hash table, which allows one value per key. Now query an entry:

(with-cabinet { :filename "test.tokyo" :mode OREADER } 
    (get-value "1"))
"one"

Tables

A table in Tokyo Cabinet can be used to store arbitrary hash maps. For example:

(def params { :filename "test-table.tokyo" :mode (+ OWRITER OCREAT) :type :table } )
(with-cabinet params
    (put-value nil { :name "John Doe" :hobbies "rowing fishing skiing" :age 28 :gender "M" })
    (put-value nil { :name "Melissa Swift" :hobbies "soccer tennis books" :age 33 :gender "F"})
    (put-value nil { :name "Tom Swift" :hobbies "inventing exploring" :gender "M" })
    (put-value nil { :name "Harry Potter" :hobbies "magic quidditch flying" :gender "M" :age 9 }))

Queries

Queries can be run, and you can use (hint) to take a look at how the query is being performed:

; show a hint and all rows matching
(defn showrows [query]
    (let [showhint (atom false)] 
        (with-query-results row query
            (when (compare-and-set! showhint false true)
                  (println "Query: " query)
                  (println "Hint: " (hint))
                  (println "Results:"))
            (println row)))
        (println))
 
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Leads to the following output:

Query:  [[:age >= 30]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Indexes

Indexes can be added with create-index (and removed with delete-index), which help optimize particular queries.

The different index types:

• INDEX-DECIMAL
• INDEX-LEXICAL
• INDEX-QGRAM

With some optional specifiers that can be added / ored in:

• INDEX-KEEP — keep the index if it already exists
• INDEX-OPTIMIZE

Running the queries again, with indexes:

; indexes are persistent
(with-cabinet params
    (create-index :hobbies INDEX-TOKEN)
    (create-index :age INDEX-DECIMAL))
 
; try the queries again with the indexes in place
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Gets the following hint:

Query:  [[:age >= 30]]
Hint:  using an index: ":age" asc (NUMGT/NUMGE)
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}
 
Query:  [[:hobbies any-token soccer]]
Hint:  using an index: ":hobbies" inverted (STROR)
token occurrence: "soccer" 1
result set size: 1
leaving the natural order
 
Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Optional Search Parameters

You can further control what’s fetched by using a number of optional specifiers in the query:

• :limit nnn — limits the number of rows returned
• :skip nnn — skips the first nnn rows
• :sort fieldname — sorts by the given field
• :order val — the specific ordering, one of SORT-NUM-ASC, SORT-NUM-DESC, SORT-TEXT-ASC, or SORT-TEXT-DESC

For example:

(with-cabinet params (with-query-results row [] (println (:name row))))    
John Doe
Melissa Swift
Tom Swift
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name]] (println (:name row))))
Harry Potter
John Doe
Melissa Swift
Tom Swift
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC]] (println (:name row))))
Tom Swift
Melissa Swift
John Doe
Harry Potter
 
(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC] [:limit 1]] (println (:name row))))
Tom Swift

Lower Level

Depending on your application, it might not be convenient to have to bracket everything with with-cabinet, since that means an open and close of the cabinet. You can also use the lower level open-cabinet and close-cabinet calls, along with the “with” statement. This is also an easier way to use it at the command line. For example:

(def test-database (open-cabinet { :filename "test-open.tokyo" :mode (+ OWRITER OCREAT) }))
(with test-database (put-value "1" "one"))
(with test-database (get-value "1"))
(with test-database (print (primary-keys)))
(close-cabinet test-database)

Miscellaneous

Use (primary-keys) to return a lazy list of primary keys.

(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREATE) :type :table }
    (print (primary-keys)))

Links

• Tokyo Cabinet
• Tokyo Cabinet / Java API