• Display is ridiculously nice
• It’s fast — subjectively, as fast as the iPad — and the extra CPU power really makes the web browsing much better.
• The bumper case is not that great — it prevents some headphones from being used (because the distance to the headphone jack is a bit longer) and also prevents you from using the 10,000 iPod USB adapters that you have lying around the house, because the slot in the case for the iPod adapter is just small enough that older adapters don’t work. As one of my friends said, “Get the Dremel!”
• I took the case off and tried to reproduce the reception problems that people had been talking about, but was unable to get it to happen.

The ominous thing was that the iPhone 4G suffered from the same wireless problems as the iPad, manifesting as a couple of symptoms: a) unable to connect unless I was really close to the router, b) and really low signal strength.

Wireless

I had been living with the wireless problems for a while, although they were extremely frustrating. Finally, a lucky break meant that I was able figure out the underlying cause (although to my chagrin I discovered later that other groups had identified this a long time ago). My iPhone 4G wouldn’t get onto the net, and even worse, every time I tried to get onto the net, it would kick my laptop off for a minute or two.

At that point, it became blindingly obvious — the phone was trying to use an expired lease, and as a result was conflicting with the laptop. Thirty seconds later I had set up static IPs for the iPad and iPhone 4G, and all my problems were solved. I could use the devices all over the house, there were no mysterious drops, signal strength was fine, and as a bonus, the mysterious wireless network failures that my laptop was having were solved.

Lessons Learned

Well, I really should have done my research — I could have found the answer to this much earlier.

Why did I think it had something to do with signal strength? I suspect that the 3.2/4.0 iOS somehow bases the signal strength indicator on DHCP status — where I was getting one bar before is now three. The alternate explanation is that I tended to use the devices in the more remote places at night, which happened to coincide with the DHCP conflict.

Create a new project directory with

Change the project.clj file that is autogenerated with:

(that is, add the mysql and incanter dependencies).

Download all the dependencies with

Start up a REPL with everything in the classpath by just using

Wow, that’s kind of nice.

Unfortunately, despite the fact that there were some good guides on how to do it, it turned out there were a few gotchas, and it took significantly longer than 10 minutes. So, in the hopes that I can save someone else the time:

Use this guide, but:

• in Windows 7, you’ll want to use the “Network and Sharing Center” (right click on your network icon) to create a new “computer-to-computer” network. This is the one the iPhone will connect to.
• When you configure Putty, you’ll want to make sure you go to the SSH/Tunnels page and a) add a dynamic tunnel to a local port (assume 8080) — you won’t need to specify a destination and can leave that blank, and b) go to the “Connection” page and specify a keepalive of 5 or so. This keeps the SSH connection between the computer and the iPhone from dying every 30 seconds.
• You cannot use Chrome right now, because it does not properly forward DNS requests to the proxy (it took me hours to figure this out). Use Firefox and specify a SOCKS v5 proxy at port 8080 (or whatever) for the proxy, but you’ll also need to make sure to go to the about:config page and change the network.proxy.socks_remote_dns to “true”.
• Doing things in a particular order makes this work better. If the 3G connection on the iPhone is not active, then there doesn’t seem to be a way to activate after it links to your ad hoc connection. So: turn off wireless, go to a web page (this activates the 3G), turn on wireless, connect to your computer. This way both the 3G and wireless connections are established. An easy way to see if the 3G is active is to run MobileTerminal (or just SSH into the iPhone) and run ipconfig; if the pdp_ip0 device is pointing to a real IP, then your 3G is active.

Man, netbooks have really come a long way. A quick run through:

Screen: 1366×768 vs. 1024×600 is almost twice as big, and makes a big difference. It’s glossy, which is kind of annoying, and I think the contrast is a little lower than on the 1000h, but nonetheless it’s really nice.

Keyboard: Full sized shift keys on the Acer! This far outweighs almost anything else. The smaller cursor control keys are kind of pain, but I’ll get used to them. Feedback is not as nice as the Asus but still pretty decent — I can type full speed on this keyboard.

Touchpad: Touchpad is good enough; at least there are two mouse buttons, which is one of the things I really wanted. It’s multitouch, but the drivers are a little choppy with the two finger scrolling — but I haven’t tried to fix it, either. The two finger to zoom in and out is surprisingly handy when browsing, to be honest, and I’m using it all the time.

Wireless: Wireless on my EEE 1000h was always a little glitchy at home (and would sometimes cause my router to reboot, for whatever reason), but was fine at work and at Starbucks. The wireless on the Acer 1410 works much better, and I haven’t had any problems.

Speaker: I frequently watch movies with the speakers on. The speakers seem to be just a touch softer than the EEE 1000h, but still pretty good. This tweaks thread has some tips on improving the volume of the speakers.

Processor: The new ULV chips are everything that Atom wanted to be. With the Acer I’ve been able to a) run Ubuntu under virtualbox, b) play full-screen 480p Hulu, and c) play full-screen Netflix, and it hasn’t stalled out on anything. Additionally, the 1000h would bog down when scrolling in Firefox (seriously), whereas this machine handles it fine.

I specifically looked for an Acer with the SU2300 (dual core) instead of the SU3500 (single core, slightly faster).

The machine only starts up the fan when doing flash video — fan noise is not too bad and significantly quieter than the EEE 1000h. The computer itself does not get very warm at all, which is nice.

VirtualBox was a must have, since I wanted to run Ubuntu but didn’t want to deal with the pain of partitioning and dealing with its peculiarities on a new machine. I had assumed that having the AMD/Intel virtualization mode was a big win, but it turns out that it really doesn’t help performance. Nonetheless, Virtualbox works fine. VT-x mode was originally turned forced off, and it was subsequently fixed in the BIOS; see this tweaks thread for tips on getting the new BIOS.

Battery Life: Hard to say, but looks like 5-7+ with this machine, whereas I was getting about 4 with the Asus 1000h.

Operating System: I run XP on most of my computers (and didn’t upgrade any to Vista), but the Acer came with Windows 7 Home Premium. It’s pretty and nice and performs well — I have no complaints.

What a difference a year makes. These new ULV chips are amazing.

I really enjoy Clojure. Everything seems so well thought out and well designed; in a lot of ways it reminds me of Python, which is three ways to ironic because back in the day I started using Python because it was very Lispy. The cycle of life turns even on computer languages.

But unfortunately, I get the sense that there’s probably only room for one JVM language to make it big. As much as I like Clojure, I think that Scala will probably be the one that succeeds. It’s got the static typing, it’s got the Java-like performance, and it’s got a syntax that’s close-ish to Java, so it fills the niche that Java fills so well: a programming language that is ideal for large groups of developers.

However, it’s early enough in the development of Clojure that there’s the sense that maybe individual efforts could make a difference. (As a counterexample, Haskell is too far along — it seems like everything has been done already, or anything that needs to be done requires a deep knowledge of category theory and the latest functional design papers.)

Assuming that I would want to help out Clojure, what could be done?

Donate? Actually, I did that. I figured that I got as many hours of enjoyment out of Clojure as I did out of a typical game, so I contributed the price of a game.

Libraries? Well, I’ve written one or two. Will probably write more.

If Clojure is going to succeed, it needs a niche, an area in which it performs so well that it blows everything else away. Concurrent programming is one possibility, but there is a lot of competition there.

I have a modest proposal:

Clojure should be more like PHP.

Yes, yes, I know that programming in PHP is socially reprehensible, just one step above people that shop without their shirt while in Walmart and car salesmen.

But what if we were to give Clojure the trappings that make starting developers try it? Metaphorically, it’s like Gulliver’s Travels; it looks like a humorous, light-hearted adventure story, but underneath it’s biting political satire and deep thoughts. We lure developers in with the possibility of an easy, light-hearted way to write web sites… and then once they’re in there, once they’ve progressed from manipulating web sites and start to realize the benefits of advanced languages, they can deal with a sane language, built on top of the JVM and with the full power of a Lisp.

So call this Gulliver.

Why Did PHP Succeed?

My impression, based on absolutely no citations or hard evidence, is that PHP became popular primarily because it is:

• Easy to start trying — start with an HTML page, and just add one statement to start playing around with PHP itself
• Easy to access data — typically, with MySQL
• Easy to deploy — just upload a file and go

So, continuing the gedankenexperiment: if we were designing something that would be very approachable and very easy to use, what would it look like?

Making Compojure More Approachable

Compojure is an excellent base to start with, but it strays away from templates, and instead tries to do everything with compojure.html, a library that allows you to form everything with vectors and lists:

Handy? Yes, if you’re a developer. But if a web designer is doing your HTML, or if you’re just starting out programming and you already know HTML, though, it seems an additional barrier to entry. So what if we turn it inside out and do it like PHP does it?

Trying something more complicated, in compojure.html this:

becomes:

A little bit longer, but treating it as HTML makes it conceptually easier for beginning programmers to understand, especially if you want to do something like changing the id or class with every row so your CSS can make it all pretty:

Doing this is compojure.html is trickier:

So, let’s lure people in with easy to use, and they can pick up vectors and S-exprs and notational convenience later. Converting a template file into the basic code is easy:

Resulting in…

Once you have the basic code generation, it’s easy enough to hook it up to a default servlet that automatically invokes it when necessary. There’s a fair bit of handwaving going on here — regexes are questionable for this, there are some interesting issues with namespaces, and it would require some method of resolving dependencies between files, but heck, it’s only 100 lines of code; it was really quick to put together and gives us a sense of whether this can work out.

So, with the source on github, your basic server becomes…

(use 'gulliver)
(defserver web-server
  {:port 8080}
  "/*"  (servlet gulliver/template-servlet))

Easy! As an test, an example file:

<?
(import 'java.util.Date)
?>

<html>
  <body>
    <h1>Delivered via Gulliver</h1>
    <p>Expressions at work: <?= (+ 2 2) ?></p>
    <p>Looping:</p>
    <ul>
    <? (dotimes [x 10] ?>
      <li><?= x ?></li>
    <? ) ?>
    </ul>

    <p>Here's a date:  <?= (Date.) ?></p>
  </body>
</html>

… does the right thing. How about a time test?

<html>
    <body>
    <h1>Starting time test...</h1>
    <? (let [ start (System/currentTimeMillis) ] ?>
    <p>Adding 1..100000:  <?= (reduce + 0 (range 1 1000001)) ?></p>
    <p>(took <?= (- (System/currentTimeMillis) start) ?> ms)</p>
    <? ) ?>
    </body>
</html>

… runs in about 50ms. PHP takes about 130ms on the same sequence on my laptop, so, while there are tons of optimizations we could do (like not doing a stat on the file every time the page is rendered (!)), it nonetheless performs well enough that maybe this is a viable option. Finally, as a special bonus, a stupid eval trick that evaluates Clojure code:

<html>
    <body>
        <? (when ((request :params) :code) ?>
        <pre style="border: 1px dotted">
        <?= (str (eval (read-string ((request :params) :code)))) ?>
        </pre>
        <? ) ?>
        <form method="post">
        <textarea name="code" rows=10 cols=80></textarea><br/>
        <input type="submit" value="Evaluate" />
        </form>
    </body>
</html>

Take that, non-dynamic languages!

Github copy of the source

.

Next part: Schema-ing Against MySQL

public void OnIdleStateChange(AccSession arg0, int arg1) {
}

public void OnInstanceChange(AccSession arg0, AccInstance arg1, AccInstance arg2, AccInstanceProp arg3) {
}

public void OnLookupUsersResult(AccSession arg0, String[] arg1, int arg2, AccResult arg3, AccUser[] arg4) {
}

public void OnSearchDirectoryResult(AccSession arg0, int arg1, AccResult arg2, AccDirEntry arg3) {
}

// ... go on like this for pages

The Java code is here, if you’re interested in the entire set of calls. Now, I didn’t care about most of those events, but I had to override them, since they didn’t have a default implementation. What made this seem painful was that I was really only interested in two of the callbacks. So I started to record an Emacs macro to convert the Java code to the equivalent Clojure proxy statement, and then I realized that I didn’t have to — I was using a Lisp.

(defmacro auto-proxy [interfaces variables & args]
  (let [defined (set (map #(str (first %)) args))
        names (fn [i] (map #(.getName %) (.getMethods i)))
        all-names (into #{} (apply concat (map names (map resolve interfaces))))
        undefined (difference all-names defined)
        auto-gen (map (fn [x] `(~(symbol x) [& ~'args])) undefined)]
    `(proxy ~interfaces ~variables ~@args ~@auto-gen)))

Auto-proxy works just like proxy, but it makes an empty implementation for any call that wasn’t defined. So, suddenly, what would have been a bunch of lines collapsed into just:

(defn create-aim-proxy []
  (auto-proxy [com.aol.acc.AccEvents] []
     (OnImReceived [session imSession participant im]
        (handle-im session imSession participant im))
     (OnStateChange [arg0 arg1 arg2]
        (handle-state-change arg0 arg1 arg2))))

Macros ftw. The nice thing about Clojure/Lisp is that it makes coding up this kind of reusable framework stuff really easy.

Clojure code density seems to be pretty good. There are a fair number of convenient shortforms in the language; for example, associative datatypes all act as a function — so given a hash map you can reference it with (my-hashmap :key). The base language itself is probably about as expressive as Python (or a bit better), but you have the added advantage of being able to use macros as needed to really get the code density up.

Nonetheless, I really wanted something like Ocaml’s / Haskell’s pattern matching; it makes some code wonderfully concise.

Accordingly, I hacked something up, based on Clojure’s built-in destructuring. Some examples:

Literal values match against the same value, while _ matches against any non-nil value (and nil matches against any nil one). Additionally, :when clauses can be used for conditional checks.

; simple recursive evaluator
(defn arithmetic [lst]
  (match lst
    v  :when (number? v)  v
    [ _ "error" _]     "error"
    [ _ _ "error"]     "error"
    [ "add" a b ]      (+ (arithmetic a) (arithmetic b))
    [ "sub" a b ]      (- (arithmetic a) (arithmetic b))
    [ "mul" a b ]      (* (arithmetic a) (arithmetic b))
    [ "div" a b ]      (/ (arithmetic a) (arithmetic b))
    [ "squared" a ]    (arithmetic ["mul" (arithmetic a) (arithmetic a)])
    _                  "error" ))

Both collections and single values can be used:

;; return signum fr a number
(defn signum [x]
  (match x
     0 0
     n :when (< n 0) -1
     _ 1))

The pattern matching is stricter than the typical destructure; whereas [ a b ] will destructure against a list of any number of elements, [ a b ] will pattern match only against a list of two elements.

(match x
    []    "empty"
    [_]   "one element"
    [a a] "two identical elements"
    [_ _] "two elements"
    _     "three or more")

If the same variable occurs in multiple locations in the parameter list, it will be checked for equality. The & tail form can be used to specify the rest of the list.

;; count identical elements in the same location in two lists:
(defn count= [ lst1 lst2 ]
  (loop [ a lst1 b lst2 count 0 ]
    (match [a b]
      [[e & at] [e & bt]]  (recur at bt (inc count))
      [[_ & at] [_ & bt]]  (recur at bt count)
      _                    count)))

Note that this is slightly more flexible than Haskell / ML, in that a variable of the same name can be multiple places in the pattern.

Defining

You can use the defnp macro to define a function that is pattern matched; it defines a function that takes one argument and has an implicit match statement. For example, the signum function can be written:

    (defnp signum
       0 0
       n :when (< n 0) -1
       _ 1)

(Thanks to Tom Faulhaber for suggesting this)

Gotchas

The Clojure destructuring will cause an exception if you try to destructure a collection type with a value.

(let [[a b] 10] a)
java.lang.UnsupportedOperationException: nth not supported on this type: Integer (NO_SOURCE_FILE:0)

… so be sure to check such cases early in your match statement, if they are possible.

How It Works

The pattern matcher uses the built-in Clojure destructuring as the main mechanism, but adorns it so that the pattern can be verified. For example, the code:

(match x [a a] "two identical" )

turns into essentially the following:

(let [ [ a g0001 & g0002 ] x ]
     (if (and (not (nil? a)) (= g0001 a) (nil? g0002)) "two identical" nil))

That is, the destructuring is done, but then the two variables are checked to make sure that they are equal, and the list is checked to make sure it is only two elements long.

Source

It’s all on Github.

Nonetheless, I ran into a situation where it seemed to be 100x slower than Java because of an erroneous assumption on my part. After stripping out and profiling and removing everything extraneous, it finally came down to array access. Ignoring the uselessness of this snippet for a while, the issue is how to translate something like the following code:

int v[] = new int[1000];
java.util.Arrays.fill(v, 0);
for (int i = 0; i < 100000; i++)
    for (int ix = 0; ix < 1000; ix++) {
        v[ix]++;
        v[ix]--;
}

… which runs in about 350ms on my Macbook. My pass at the Clojure equivalent, with the type hinting:

;; this is 100x slower than the equivalent in Java
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]
        (aset-int v ix (unchecked-add (int 1) (aget v ix)))
        (aset-int v ix (unchecked-subtract (int 1) (aget v ix)))))))

… which takes about 40692ms — more than 100x slower. What the hell? After many hours of Googling, I finally discovered from the comments on this blog post that aset-int is slower than aset, which I don’t recall seeing anywhere else. Does it really make a difference?

;; much faster
(defn useless-array-manipulation []
  (let [v (int-array 1000)]
    (java.util.Arrays/fill v 0)
    (dotimes [_ (int 100000)]
      (dotimes [ix (int 1000)]>
        (aset v ix (unchecked-add (aget v ix) (int 1)))
        (aset v ix (unchecked-subtract (aget v ix) (int 1)))))))

… runs in about 1000ms, or about 3x slower than the Java variant, and is definitely something that I can live with. I had been assuming that aset-int was of course faster because it was more specific, but in fact (aset v ix (int val)) is much faster than (aset-int v ix val).

At any rate, I made a small, thin layer around the Tokyo Cabinet API, and put it on Github. Another thin wrapper can be found at this blog.

Copy of the README is below (the ultimate in lazy!).

Introduction

This is a simple interface to the Tokyo Cabinet libraries. Tokyo Cabinet is a very cool, very high performing key-value store. This library supports table mode, which essentially means that arbitrary hashmaps can be stored in the cabinet.

Note that this is appropriate for local storage only — if you’re looking to share a Tokyo Cabinet to multiple computers, you actually want Tokyo Tyrant.

Basic Usage

The with-cabinet call creates/opens a cabinet and allows the use of the various access routines within the scope of the call. For example, here’s how to create a cabinet with three entries.

(ns user (:use tokyo-cabinet))  ;; bring into our namespace

(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREAT) }
    (doseq [[name val] [["1" "one"] ["2" "two"] ["3" "three"]]]
        (put-value name val)))

This creates a Tokyo Cabinet hash table, which allows one value per key. Now query an entry:

(with-cabinet { :filename "test.tokyo" :mode OREADER }
    (get-value "1"))
"one"

Tables

A table in Tokyo Cabinet can be used to store arbitrary hash maps. For example:

(def params { :filename "test-table.tokyo" :mode (+ OWRITER OCREAT) :type :table } )
(with-cabinet params
    (put-value nil { :name "John Doe" :hobbies "rowing fishing skiing" :age 28 :gender "M" })
    (put-value nil { :name "Melissa Swift" :hobbies "soccer tennis books" :age 33 :gender "F"})
    (put-value nil { :name "Tom Swift" :hobbies "inventing exploring" :gender "M" })
    (put-value nil { :name "Harry Potter" :hobbies "magic quidditch flying" :gender "M" :age 9 }))

Queries

Queries can be run, and you can use (hint) to take a look at how the query is being performed:

; show a hint and all rows matching
(defn showrows [query]
    (let [showhint (atom false)]
        (with-query-results row query
            (when (compare-and-set! showhint false true)
                  (println "Query: " query)
                  (println "Hint: " (hint))
                  (println "Results:"))
            (println row)))
        (println))

(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Leads to the following output:

Query:  [[:age >= 30]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order

Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Query:  [[:hobbies any-token soccer]]
Hint:  scanning the whole table
result set size: 1
leaving the natural order

Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Indexes

Indexes can be added with create-index (and removed with delete-index), which help optimize particular queries.

The different index types:

• INDEX-DECIMAL
• INDEX-LEXICAL
• INDEX-QGRAM

With some optional specifiers that can be added / ored in:

• INDEX-KEEP — keep the index if it already exists
• INDEX-OPTIMIZE

Running the queries again, with indexes:

; indexes are persistent
(with-cabinet params
    (create-index :hobbies INDEX-TOKEN)
    (create-index :age INDEX-DECIMAL))

; try the queries again with the indexes in place
(with-cabinet params
    (showrows [[:age ">=" 30]])
    (showrows [[:hobbies "any-token" "soccer"]]))

Gets the following hint:

Query:  [[:age >= 30]]
Hint:  using an index: ":age" asc (NUMGT/NUMGE)
result set size: 1
leaving the natural order

Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Query:  [[:hobbies any-token soccer]]
Hint:  using an index: ":hobbies" inverted (STROR)
token occurrence: "soccer" 1
result set size: 1
leaving the natural order

Results:
{:gender F, :hobbies soccer tennis books, :name Melissa Swift, :age 33}

Optional Search Parameters

You can further control what’s fetched by using a number of optional specifiers in the query:

• :limit nnn — limits the number of rows returned
• :skip nnn — skips the first nnn rows
• :sort fieldname — sorts by the given field
• :order val — the specific ordering, one of SORT-NUM-ASC, SORT-NUM-DESC, SORT-TEXT-ASC, or SORT-TEXT-DESC

For example:

(with-cabinet params (with-query-results row [] (println (:name row))))
John Doe
Melissa Swift
Tom Swift
Harry Potter

(with-cabinet params (with-query-results row [[:sort :name]] (println (:name row))))
Harry Potter
John Doe
Melissa Swift
Tom Swift

(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC]] (println (:name row))))
Tom Swift
Melissa Swift
John Doe
Harry Potter

(with-cabinet params (with-query-results row [[:sort :name] [:order SORT-TEXT-DESC] [:limit 1]] (println (:name row))))
Tom Swift

Lower Level

Depending on your application, it might not be convenient to have to bracket everything with with-cabinet, since that means an open and close of the cabinet. You can also use the lower level open-cabinet and close-cabinet calls, along with the “with” statement. This is also an easier way to use it at the command line. For example:

(def test-database (open-cabinet { :filename "test-open.tokyo" :mode (+ OWRITER OCREAT) }))
(with test-database (put-value "1" "one"))
(with test-database (get-value "1"))
(with test-database (print (primary-keys)))
(close-cabinet test-database)

Miscellaneous

Use (primary-keys) to return a lazy list of primary keys.

(with-cabinet { :filename "test.tokyo" :mode (+ OWRITER OCREATE) :type :table }
    (print (primary-keys)))

Links

• Tokyo Cabinet
• Tokyo Cabinet / Java API

So, I wrote a tiny little program to help facilitate using git and Wordpress together through the XMLRPC API, and it’s helped me write… well, if not more than at least less painfully.

Setting Up

Assuming that your blog is set up at http://www.yourblog.com (you can create an account at wordpress.com to test this out), all you’ll need to do is:

-- make a directory for the blog
mkdir blog
chdir blog

-- download everything
python wp.py download --user=yourname --password=yourpass --url=http://www.yourblog.com/xmlrpc.php
(wait a bit)
git init
git add .
git commit -m "first version"

-- now set up so we don't have to specify --user, --password, and --url every time (optional)
git wp config wp.url http://www.yourblog.com/xmlrpc.php
git wp config wp.user yourname
git wp config wp.password yourpass

The files are downloaded in the appropriate YYYY/MM directories, with the draft directory being used for all of your unpublished drafts.

All the drafts are stored in plain text, but you’ll see some lines starting with periods — these are various Wordpress variables that are associated with the file. You can change them, as well; for example, to change the title of the post, just change the line that begins with “.title”.

Seeing What’s Different

You can use the status command to see differences between the local file system and your blog.

python wp.py status

Updating From The Blog

If you’ve made changes through the web interface and you’d like to bring them down, you don’t have to download everything again, but can instead just update.

python wp.py update

Posting/Editing

If you’d like to edit a post, just edit it, and then use wp post to push it back to the blog:

python wp.py post changed-file

To create a new post, just create a new file in the draft folder — I like to specify the .title and .wp_slug parameters, as well — and then post it. You can also publish it by changing the .post_status line from draft to publish.

Gotchas

While this program requires git, it doesn’t automatically check anything in — so you’ll need to make sure you do git add / git commit or whatnot as necessary.

There are also some gotchas due to the fact that the filename can change on you. There are cases where the filename that will be brought down is different then the one that you send up:

• You post a file without a .title or .wp_slug line
• You post a file with a different file name than the slug that is generated (i.e., “my-first-draft” when the title is actually “my final draft”)
• Something moves from “draft” to “publish”

When you see a message of the form “changed: fn1 -> fn2″, it means that a rename has occurred, and you’ll need to do the git rm/git add or git mv by hand.

Source

Source is available on github.

I needed to interface with a bunch of data that had PHP wrapper classes, and needed a quick way of being able to interface with PHP from Python. (At this point, you might find it hard to believe that the PHP wrapper classes were worth trying to reuse, but the wrapper classes took care of partitioning and caching and everything, so it seemed silly to not reuse them). I considered using the PHP API and ctypes to make a Python to PHP bridge , but then decided that was entirely too much work, so I just hacked together a stupid class that invoked PHP with arbitrary code and returned it in a number of ways.

Basic usage

Create the PHP class (giving it an optional prefix and postfix — prefix is the most useful, as it allows you to specify requires), and then invoke a code block with get_raw, get, or get_one. Note that every “call” into PHP will create and destroy a PHP process.

Some examples:

Reading raw input

php = PHP("require '../code/private/common.php';")
code = """for ($i = 1; $i <= 10; $i++) { echo "$i\n"; }"""
print php.get_raw(code)

will output a string containing the numbers “1″..”10″

Reading one big JSON value

php = PHP("require '../code/private/common.php';")
code = """
$a = array();
for ($i = 1; $i <= 10; $i++) { $a[] = $i; }
echo json_encode($a);"""
print php.get(code)

This code would return a Python list with the numbers 1..10.

Reading many JSON values

    php = PHP("require '../code/private/common.php';")
    code = """
    for ($i = 1; $i <= 10; $i++) {
        echo json_encode(array($i => $i * $i)) . "\n";
    }"""
    for row in php.get_one(code):
        print row

Given a PHP snippet that returns one JSON value per line, iterate through them as Python values.

The actual code (also on Github):

import popen2
import simplejson as json

class PHP:
    """This class provides a stupid simple interface to PHP code."""

    def __init__(self, prefix="", postfix=""):
        """prefix = optional prefix for all code (usually require statements)
        postfix = optional postfix for all code
        Semicolons are not added automatically, so you'll need to make sure to put them in!"""

        self.prefix = prefix
        self.postfix = postfix

    def __submit(self, code):
        (out, inp) = popen2.popen2("php")
        print >>inp, "<? "
        print >>inp, self.prefix
        print >>inp, code
        print >>inp, self.postfix
        print >>inp, " ?>"
        inp.close()
        return out

    def get_raw(self, code):
        """Given a code block, invoke the code and return the raw result as a string."""
        out = self.__submit(code)
        return out.read()

    def get(self, code):
        """Given a code block that emits json, invoke the code and interpret the result as a Python value."""
        out = self.__submit(code)
        return json.loads(out.read())

    def get_one(self, code):
        """Given a code block that emits multiple json values (one per line), yield the next value."""
        out = self.__submit(code)
        for line in out:
            line = line.strip()
            if line:
                yield json.loads(line)

But, git had everything still around — it turns out to be pretty easy to get it back. The magic command turned out to be git reflog.

So I had lost everything in HEAD@{1}, but you can get it back by just checking out that particular commit.

Note that if I didn’t need to fiddle around with stuff, but just had wanted everything in the commit, I could just have merged the entire commit:

… and that would have brought everything back to the state I needed.

data S = S { name :: String } deriving (Show)
firstname s = (words (name s))!!0
lastname  s = (words (name s))!!1

But, dammit, you want to invoke it like you would in C++. So define a function:

*Main> let (-->) x f = f x
*Main> let test = S "George Washington"
*Main> test --> firstname
"George"
*Main> test --> lastname
"Washington"

(It’s tempting to use `.`, but it conflicts with the Prelude. Also note that you could define it as “(–>) = flip ($)”). But what if it takes more than one parameter?

*Main> let flip_concat t s = intercalate t $ reverse.words $ name s
*Main> test --> flip_concat ", "
"Washington, George"
*Main> test --> flip_concat(", ")
"Washington, George"

… although you might not like that anything can be applied to (–>).

*Main> [1,2,3] --> length
3

class Deref a where
    (-->) :: a -> (a -> b) -> b
    x --> f = f x 

instance Deref S

*Main> [1,2,3] --> length

<interactive>:1:0:
    No instance for (Deref [t])
      arising from a use of `-->' at <interactive>:1:0-17
    Possible fix: add an instance declaration for (Deref [t])
    In the expression: [1, 2, 3] --> length
    In the definition of `it': it = [1, 2, 3] --> length

You can even use tuple passing to make it look even more like a typical call.

*Main> let pretty (pre, mid, post) s = pre ++ (firstname s) ++ mid ++ (lastname s) ++ post
*Main> test --> pretty ("<", ", ", ">")
"<George, Washington>"

… although that makes it harder to curry.

import qualified Data.Map as Map
import Data.Maybe
import Network.XmlRpc.Client
import Network.XmlRpc.Internals

server = "http://yourserver.wordpress.com/xmlrpc.php"

-- extract multiple posts from the XML response
extract :: Value -> [Map String Value]
extract xmlresp =
    let ValueArray rs = xmlresp in
    map (\v -> case v of
                  ValueStruct vs -> Map.fromList vs
                  _              -> Map.fromList []) rs

getRecentPosts :: Int -> [Char] -> [Char] -> Int -> IO Value
getRecentPosts = remote server "metaWeblog.getRecentPosts"

-- print out the five most recent posts
main = do result <- getRecentPosts 1 "yourname" "yourpass" 5
          let posts = extract result
          mapM_ (\p -> print $ fromJust $ Map.lookup "title" p) posts

{-# OPTIONS -XBangPatterns #-}
import IO
import System
import Data.Word
import Data.Array.Unboxed
import Control.Monad.ST
import Data.Array.ST
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C

counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   -- collate character counts here
                   return arr
counts !line = runSTUArray (counts' line)

hit !line = let freq = counts line in freq!65 == 10

main =
    do args <- getArgs
       f <- openFile "wordlist" ReadMode
       text <- B.hGetContents f
       print $ length $ filter hit (C.lines text)

This is fast (0.07s on 300K test file, compiled with -O2 on a Macbook Pro), but there’s no actual collation going on. It suddenly becomes two magnitudes slower as soon as you start to do anything based on the line:

-- version 1 : 9.5 seconds
counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   readArray arr (B.head line) >>= \v -> writeArray arr (B.head line) (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

.. equals 9.54 seconds just when you collate the first character of the string, and led me to believe that ByteStrings were slow, especially since a constant change to the array was fast (0.07 seconds):

-- version 2 : 0.07 seconds
counts' !line = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   readArray arr 0 >>= \v -> writeArray arr 0 (v+1)
                   return arr

(Here I’ll elide the hours of me messing around with profiling and whatnot to try and figure out why B.head was so slow, or how to make B.foldl’ update a state variable, or sprinkling strictness bangs all over the place, or the other tons of false trails that I went down.)

It turns out that the slow portion in all of this is the newArray, not the character collation itself, which can be seen in the profile if the array creation is moved into its own routine and we look at the profile:

-- version 3
-- compile with: ghc -package bytestring stuarray.hs -prof -auto-all -O2 -o stuarray.out
-- run with: ./stuarray.out +RTS -p
initial_array = do arr <- newArray (0,255) 0 :: ST s (STUArray s Word8 Int)
                   return arr

counts' !line = do arr <- initial_array
                   let ix = B.head line
                   readArray arr ix >>= \v -> writeArray arr ix (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

… and the relevant part of the profile:

MAIN                     MAIN                     1           0   0.0    0.0   100.0  100.0
 main                    Main                   240           1   0.7    0.3   100.0  100.0
  hit                    Main                   242      335075   0.0    0.0    99.3   99.7
   counts                Main                   243      335075   0.0    0.2    99.3   99.7
    counts'              Main                   244      335075   0.0    0.0    99.3   99.5
     initial_array       Main                   245      335075  99.3   99.5    99.3   99.5

Which brings up two interesting points:

• The Haskell optimizer was so smart that it was able to figure out that the code in version 2 built a constant array… and so only called the newArray once. (Confirmed with a profile).
• New array creation seems to be so slow that it dominates a benchmark that has file I/O.

The Ocaml version took about 0.720 seconds (Ocaml version below), compared to Haskell time of about 8.5 seconds. Reducing the array size to 128 in both cases reduced it to 0.112 seconds for Ocaml and 4.3 seconds in Haskell.

Note the discontinuity in Ocaml between 255 and 256 elements, which I find interesting. The nice people in #haskell suggested that I stop constructing/destructing the array and instead just null it out between each line, and it turned out that the best way to do that was just to thaw a starter array.

-- version 4: 0.11 seconds
initial_array = listArray (0,255) (repeat 0) :: UArray Word8 Int

counts' !line = do arr <- thaw initial_array
                   let ix = B.head line
                   readArray arr ix >>= \v -> writeArray arr ix (v+1)
                   return arr
counts !line = runSTUArray (counts' line)

This is still kind of strange to me (because an object is still getting constructed/destructed — thaw guarantees a copy) but you can’t argue with a performance increase. (Interestingly, using Array.copy or Array.fill in Ocaml is slower than just using Array.make). Final times? 0.11 seconds in Haskell vs. the best of 0.720 second UPDATED: 0.12 seconds in Ocaml… and the Haskell version is just 50% slower than a C implementation with a gratuitous calloc instead of a memset.

UPDATE: An pointed out that zeroing the Ocaml array with a for loop is much faster than Array.copy or Array.fill, bringing it to about the same speed as Haskell.

Lessons learned?

• Performance is a treacherous mistress
• The Haskell optimizer is awesome but you have to be wary when trying to narrow down performance problems
• #haskell is always full of useful suggestions
• Array creation seems to be slow enough that alternatives should be explored.

open Char
open String

let all_lines fn filename =
    let chan = open_in filename in
        try
            while true do
                let line = input_line chan in
                    fn line
            done
        with End_of_file ->
            close_in chan

let initial_array = Array.make 256 0

let count line =
    let freq = initial_array in
    let ix = int_of_char line.[0] in
        for i = 0 to 255 do freq.(i) <- 0 done;
        freq.(ix) <- freq.(ix) + 1;
        freq

let hit line =
    let freq = count line in
        freq.(65) = 10

let _ =
    let linecount = ref 0 in
        all_lines (fun line -> if hit line then linecount := !linecount + 1 else ()) "wordlist";
        print_int !linecount

import IO
import System
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C

main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f
          print $ length $ filter ((==) searchfor) (C.lines text)

To search a smallish list of about 300K words takes 0.040 seconds on my computer, compared to 0.200 seconds for Python and 0.210 seconds for a naive Haskell implementation that is not using ByteStrings. However, let’s just add lowercase to the equation:

import IO
import System
import Data.Char
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C

main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f
          print $ length $ filter (\x -> (C.map toLower x) == searchfor) (C.lines text)

Suddenly, the ByteString version becomes about 30% slower than the naive version — 0.337 seconds vs. 0.251 seconds — and is even slower than the Python version. What the heck is going on here? Trying an empty map (i.e., C.map id x) resulted in something fast, so I’m suspecting that the lowercase function itself is slow.

Unfortunately, there doesn’t seem to be a lowercase available in ByteString; at the moment it seems that you need to set up your own ctype table and use that.

import IO
import System
import Data.Char
import Data.Word
import Data.Array.Unboxed
import qualified Data.ByteString as B
import qualified Data.ByteString.Internal as BI
import qualified Data.ByteString.Char8 as C

ctype_lower = listArray (0,255) (map (BI.c2w . toLower) ['\0'..'\255']) :: UArray Word8 Word8
lowercase = B.map (\x -> ctype_lower!x)

main = do args <- getArgs
          let searchfor = C.pack $ head args
          f <- openFile "wordlist" ReadMode
          text <- B.hGetContents f
          print $ length $ filter (\x -> (lowercase x) == searchfor) (C.lines text)

… which turns out to run really quickly at 0.070 seconds, about the same as a C program doing the same task.

Update: See dons comments below — Char is operating on Unicode, which makes it slow. I wonder if a ctype.h-type library for ByteString makes sense?

Installing MissingPy

I had an issue using Cabal to install MissingPy into my GHC 6.10 installation (Mac OS X). I went into the directory that had been unpacked and modified Setup.hs:

import Distribution.PackageDescription
import Distribution.PackageDescription.Parse  -- added this line
import Distribution.Simple

A cabal configure, build, and install of the package then installed everything.

Basic Usage

It’s easiest to play around with MissingPy in ghci. Before you do anything you’ll need to initialize the interpreter with py_initialize and import any necessary modules with pyImport. You can test by using pyRun_SimpleString:

# :m Python.Interpreter Python.Objects Python.Utils Python.Exceptions
# py_initialize
# pyRun_SimpleString "print 'hello'"
hello

The simplest useful way to execute python code is to use pyRun_String. For example:

-- add 1 and 2 in Python
-- can also use Py_file_input or Py_single_input for different handling of the context
addem = pyRun_String "1+2" Py_eval_input [] >>= fromPyObject :: IO Integer

There are also the callByName and pyObject_Call methods for more complicated scenarios.

callByName :: String            -- ^ Object\/function name
           -> [PyObject]        -- ^ List of non-keyword parameters
           -> [(String, PyObject)] -- ^ List of keyword parameters
           -> IO PyObject

pyObject_Call :: PyObject       -- ^ Object to call
              -> [PyObject]     -- ^ List of non-keyword parameters (may be empty)
              -> [(String, PyObject)] -- ^ List of keyword parameters (may be empty)
              -> IO PyObject    -- ^ Return value

-- example 1: create an IMAP instance
imap <- callByName "imaplib.IMAP4_SSL" [imap_hostname] []

-- call "foo.bar(p1,p2)"
bar <- getattr foo "bar"
result <- pyObject_Call bar [p1,p2] []

Manipulating Python Objects

The fromPyObject and toPyObject handle marshaling of objects from Haskell to Python and vice versa. There are some handy functions for doing common operations on Python objects that are especially useful from the REPL: reprOf (like Python repr), strOf (Python str), showPyObject (shows the type and repr), and dirPyObject (like the Python dir()).

# g <- pyRun_String "(1,2,'hello')" Py_eval_input []
# showPyObject g
"<type 'tuple'>: (1, 2, 'hello')"
# reprOf g
"(1, 2, 'hello')"
# strOf g
"(1, 2, 'hello')"
# dirPyObject g
["__add__","__class__","__contains__","__delattr__","__doc__","__eq__","__ge__",
"__getattribute__","__getitem__","__getnewargs__","__getslice__","__gt__",
"__hash__","__init__","__iter__","__le__","__len__","__lt__","__mul__","__ne__",
"__new__","__reduce__","__reduce_ex__","__repr__","__rmul__","__setattr__","__str__"]

Dealing with Data

Tuples look like lists when they are returned. There is no conversion of tuples in the toPyObject call, so you’ll need to just convert a list and then convert it to a tuple with pyList_AsTuple. Passing “None,” “True,” or “False” is trickier; the only way I found to do it was create an instance of the value and use it in subsequent calls.

# a
[1,2]
# b <- toPyObject a
# reprOf b
"[1L, 2L]"
# pyList_AsTuple b >>= reprOf
"(1L, 2L)"

-- get "True," "False," and "None"
none <- pyRun_String "None" Py_eval_input []
pyTrue <- pyRun_String "True" Py_eval_input []
pyFalse <- pyRun_String "False" Py_eval_input []

-- try it out
# r <- pyRun_String "1==1" Py_eval_input []
# r == pyTrue
True

Passing Haskell Objects

If you want to call into functions without having to deal with Python objects, there are a couple of calls that automatically marshal the objects back and forth for you (all suffixed with “Hs”), although they turn out to be less than useful in most situations, since a) the parameters passed in must all be the same type, and b) extracting most return objects from Python is painful.

Exceptions

Python exceptions can be caught and handled; probably the easiest default way to do this is to use handlePy in conjunction with the ex2ioerr handler, which will automatically print the exception and fail.

addem = handlePy exc2ioerror $
           do r <- pyRun_String "1/0" Py_eval_input []
              fromPyObject r :: IO Integer

# addem
*** Exception: user error (Python <type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero)

another line added
another line added

Joining a string list

-- Python:  ",".join(lst)
-- either...
import Data.List
intercalate "," lst
-- or...
import Data.List
concat (intersperse "," lst)

Splitting on a string

-- a.split(ss)
-- You might need to get Data.List.Split from Cabal
import Data.List.Split
splitOn ss a
-- using regexes
import Text.Regex
splitRegex (mkRegex ss) a

Does the string start with the prefix?

-- Python: a.startswith(b)
import Data.List
isPrefixOf b a
-- ByteString
import Data.ByteString
isPrefixOf b a

Does the string end with the suffix?

-- Python: a.endswith(b)
import Data.List
isSuffixOf b a
-- ByteString
import Data.ByteString
isSuffixOf b a

Does one string contain another?

-- Python: a.find(b) != -1
import Data.List
isInfixOf b a
-- ByteString
import Data.ByteString
isInfixOf b a
-- ByteString
import Data.ByteString
findSubstring b a -- returns Just int or Nothing

Reverse a string

-- Python: a[::-1]
reverse a
-- ByteString
import Data.ByteString
reverse bs

Access one character

-- Python: a[pos]
a !! pos    -- slow O(n)
-- ByteString
import Data.ByteString
index a pos

Where does one string contain another?

-- Python: a.find(b)
import Data.ByteString
findSubstring a b

Replace substrings

-- Python: b = a.replace(src,dest)
import Text.Regex
b = subRegex (mkRegex src) a dest

Replace regex, calling function on substitution

-- Python: b = re.sub("[0-9]+", lambda x: str(int(x.group(0)) + 1), a)
import Text.Regex (matchRegexAll, mkRegex)

subRegexFn fn re s = concat $ reverse $ sub s []
    where sub s accum = case matchRegexAll re s of
                            Nothing -> s:accum
                            Just (pre, mid, post, _) -> sub post $ (fn mid):pre:accum 

b = subRegexFn (read . (+ 1) . show) "[0-9]+" a

Strip leading/trailing spaces

-- Python: a.strip()
import Data.Char (isSpace)
trim      :: String -> String
trim      = f . f
   where f = reverse . dropWhile isSpace

Loop through every line in stdin

-- Python:  loop through every line in stdin, applying a function
-- for line in sys.stdin:
--      print fn(line)
let main = interact (unlines . map fn . lines)
-- or...
let main = do { a <- stdin; mapM_ print (map fn (lines a)); }

Presume that you’ve got a function that, given a list, returns a list of booleans corresponding to whether the value was the first occurrence in the list. In Python it would look something like this:

def is_first(lst):
    found = set()
    r = []
    for item in lst:
        r.append(item not in found)
        found.add(item)
    return r

For example:

>>> is_first([1,2,3,1,4])
[True, True, True, False, True]

The most simplistic way of converting this to Haskell would require using recursion for the looping, resulting in the following (non-tail-recursive) function:

first  lst = first' lst []
first' [] _ = []
first' (h:t) state =
       if h `elem` found then False:(first' t state)
       else True:(first' t (h:state))

The first’ function needs to “carry around” state as it loops around. Now, say that the lookup function is more complicated than we thought, so we break it out into a separate function. We have to pass around the state to all functions that access it, resulting in:

first  lst = first' lst []
first' [] _ = []
first' (h:t) state =
       if (isfound h state) then False:(first' t state)
       else True:(first' t (h:state))

isfound h state = h `elem` state

Since isfound references the state, then it needs to be passed in explicitly.

I think that my main problem with understanding the State monad was that I kept wanting to treat the state as a global variable; so, I was thinking of “s <- get" and "put s" as references to a global stored somewhere, which left me with the question of "where is the initial put that sets the state?" Instead, though, you should think of "s <- get" and "put s" as references to an implicitly passed parameter.

The State Monad allows this particular pattern to be changed so that the state doesn't need to be explicitly passed. Rewriting:

state_first lst = evalState (state_first' lst) []
state_first' []    = return []
state_first' (h:t) = do state <- get
                        let found = h `elem` state
                        put (if found then state else (h:state))
                        rest <- state_first' t
                        return $ (not found):rest

Basically, given a Haskell function, you can convert it into the state monad by:

• Removing the explicitly passed state
• Changing all references to the state parameter to "param <- get"
• Instead of recursively descending, use "put new-state-value"
• Calling a state returns using "<-" to 'strip' the value

Now, this doesn't seem more elegant, but note that if you have to break functionality out, the state parameter no longer needs to be explicitly passed:

state_first lst = evalState (state_first' lst) []
state_first' []    = return []
state_first' (h:t) = do state <- get
                        found <- isfound h
                        put (if found then state else (h:state))
                        rest <- state_first' t
                        return $ (not found):rest

isfound h = do state <- get
               return $ h `elem` state

UPDATED: following a question on #haskell (which was very friendly, especially to a complete newcomer), the following more elegant ways of solving this came up:

-- using mapAccum
first lst = snd $ mapAccumL first' [] lst
    where first' acc x = (if found then acc else x:acc, not found)
              where found = x `elem` acc

-- with mapM (not exactly equivalent to example)
first lst = evalState first' []
    where first' = mapM (\a -> do state <- get; put (a:state); return $ not (a `elem` state)) lst

-- with mapM, equivalent to example
first lst = evalState first' []
    where first' = mapM isfound lst
          isfound a = do state <- get
                         let found = a `elem` state
                         put (if found then state else (a:state))
                         return $ not found

The old drive was in /dev/sdb1, while the new drive was in /dev/sdc1, so it was the work of a few moments to type:

See what I did there? Yup, I copied the new drive onto the old drive, and, yes, I sometimes do incredibly stupid things. I stopped it just a couple of seconds later, but I had lost the first 100MB of my drive.

Most of my stuff is backed up to at least two places, but there was probably about 200GB of stuff that I had lost, non-critical non-financial files that I would have preferred to get back. I eventually got it all back in the hackiest way possible.

How did I get it back?

1. Ran mke2fs -n to find some good superblocks.

$ mke2fs -n /dev/sdc1
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30515200 inodes, 122035756 blocks
6101787 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
3725 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000

2. Verified that the superblocks were good.
All of the “good” superblocks should be identical.

dd if=/dev/sdc1 skip=32768 count=1 bs=4096 | hex | head
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00194333 s, 2.1 MB/s
0x00000000: 00 40 a3 03 2c 1e 46 07 - 05 9f 12 00 d5 c4 28 07 .@..,.F.....(.
0x00000010: f5 3f a3 03 00 00 00 00 - 02 00 00 00 02 00 00 00 .?..............
0x00000020: 00 80 00 00 00 80 00 00 - 00 40 00 00 00 00 00 00 .........@......
0x00000030: 4c db 85 47 00 00 15 00 - 53 ef 00 00 01 00 00 00 L..G....S.......
0x00000040: c2 6f 0d 3a 00 4e ed 00 - 00 00 00 00 01 00 00 00 .o.:.N..........
0x00000050: 00 00 00 00 0b 00 00 00 - 80 00 01 00 04 00 00 00 ................
0x00000060: 02 00 00 00 01 00 00 00 - e6 9f 75 ff 9c 5d 49 0a ..........u..]I.
0x00000070: 95 cb a2 fb e0 39 ce 9c - 00 00 00 00 00 00 00 00 .?9..........
0x00000080: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
0x00000090: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................

I compared it with one of the later superblocks. I was lucky and had only lost the last one.

2. Get a temporary root directory onto the drive. For my particular case, I copied the start of an identically sized drive to the damaged drive so that I could get enough of a file system to work with.

3. Scan the drive and figure out what directories you want to try to get back. I ended up writing a small Python program to scan the drive directly, finding the subdirectories that I wanted.

import struct
import sys

dsk = file(sys.argv[1], "rb")
blocksize = 4096
block = 0

# see if this block could potentially be a directory
def isvalid(bytes):
    if len(bytes) < 8:
        print "error in block", block

    (inode, reclen, namelen, ftype) = struct.unpack("IHBB", bytes[0:8])

    if namelen != 1 or ftype != 2 or bytes[8] != '.':
        return False

    (inode, newreclen, namelen, ftype) = struct.unpack("IHBB", bytes[reclen:reclen+8])

    if namelen != 2 or ftype != 2 or bytes[reclen+8] != '.' or bytes[reclen+9] != '.':
        return False

    return True

# list some names in the given directory block
def names(bytes):
    ix = 0
    while ix < len(bytes):
        (inode, reclen, namelen, ftype) = struct.unpack("IHBB", bytes[ix:ix+8])
        name = bytes[ix+8:ix+8+namelen]
        yield name
        ix += reclen

while True:
    bytes = dsk.read(blocksize)

    if len(bytes) == 0: break

    if isvalid(bytes):
        print "block #", block
        print list(names(bytes))

    block += 1

You’re looking for directories that are as high level as possible — in my case, I had a directory to_archive that contained all the things that I hadn’t redundantly backed up. It was at block 22839809.

4. Given a block number, it’s possible to convert it into an inode… but it’s much easier to just go to the directory in question and take a look at the inode for it.

The first four bytes are the inode. Copy them down.

5. I found the temporary root directory that I copied from the other drive.

This directory is mostly pointing into garbage, but we just want to point one directory to our stuff.

6. Now we want to point the inode of a directory to the one that contains our files.

The root directory looks something like this.

0x00000000: 02 00 00 00 0c 00 01 02 - 2e 00 00 00 02 00 00 00 ................
0x00000010: 0c 00 02 02 2e 2e 00 00 - 0b 00 00 00 14 00 0a 02 ................
0x00000020: 6c 6f 73 74 2b 66 6f 75 - 6e 64 00 00 01 e0 0a 00 lost+found......
0x00000030: 0c 00 03 02 76 61 72 00 - 01 40 17 00 0c 00 03 02 ....var..@......
0x00000040: 65 74 63 00 01 e0 0c 00 - 10 00 05 02 6d 65 64 69 etc.........medi
0x00000050: 61 00 00 00 0c 00 00 00 - 10 00 05 07 63 64 72 6f a...........cdro
0x00000060: 6d 00 00 00 01 00 12 00 - 0c 00 03 02 62 69 6e 00 m...........bin.
0x00000070: 01 60 20 00 0c 00 04 02 - 62 6f 6f 74 01 40 13 00 .` .....boot.@..
0x00000080: 0c 00 03 02 64 65 76 00 - 01 20 1b 00 0c 00 04 02 ....dev.. ......
0x00000090: 68 6f 6d 65 01 40 03 00 - 0c 00 03 02 6c 69 62 00 home.@......lib.

The name of a file is preceded by a a) one byte type, b) a one byte name length, c) two bytes of record length, and d) four bytes of an inode. In this case, we’re going to change lost+found from inode 0×0b to the one that we got in step 4. Easiest way is to use dd and tweak:

7. Now, mount it and cross your fingers.

In my case, I was lucky — I was able to get everything back in the directories that I cared about.