Recovering A Trashed Ext3 System

tim — Tue, 10 Feb 2009 07:56:16 +0000

My 500GB backup drive started making horrible, horrible noises when spinning up, akin to a lawnmower trying to mow a rock, so I decided that it was time get another backup drive so that I would be safe. I picked up another drive at Fry’s (not a Maxtor this time, thank you very much; I haven’t had much luck with Maxtor) and hooked them both up to my computer so that I could completely copy from the old drive to the new drive.

The old drive was in /dev/sdb1, while the new drive was in /dev/sdc1, so it was the work of a few moments to type:

dd if=/dev/sdc1 of=/dev/sdb1

See what I did there? Yup, I copied the new drive onto the old drive, and, yes, I sometimes do incredibly stupid things. I stopped it just a couple of seconds later, but I had lost the first 100MB of my drive.

Most of my stuff is backed up to at least two places, but there was probably about 200GB of stuff that I had lost, non-critical non-financial files that I would have preferred to get back. I eventually got it all back in the hackiest way possible.

How did I get it back?

1. Ran mke2fs -n to find some good superblocks.

$ mke2fs -n /dev/sdc1
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30515200 inodes, 122035756 blocks
6101787 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
3725 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
	102400000

2. Verified that the superblocks were good. All of the “good” superblocks should be identical.

dd if=/dev/sdc1 skip=32768 count=1 bs=4096 | hex | head
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00194333 s, 2.1 MB/s
0x00000000: 00 40 a3 03 2c 1e 46 07 - 05 9f 12 00 d5 c4 28 07 .@..,.F.....(.
0x00000010: f5 3f a3 03 00 00 00 00 - 02 00 00 00 02 00 00 00 .?..............
0x00000020: 00 80 00 00 00 80 00 00 - 00 40 00 00 00 00 00 00 .........@......
0x00000030: 4c db 85 47 00 00 15 00 - 53 ef 00 00 01 00 00 00 L..G....S.......
0x00000040: c2 6f 0d 3a 00 4e ed 00 - 00 00 00 00 01 00 00 00 .o.:.N..........
0x00000050: 00 00 00 00 0b 00 00 00 - 80 00 01 00 04 00 00 00 ................
0x00000060: 02 00 00 00 01 00 00 00 - e6 9f 75 ff 9c 5d 49 0a ..........u..]I.
0x00000070: 95 cb a2 fb e0 39 ce 9c - 00 00 00 00 00 00 00 00 .?9..........
0x00000080: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
0x00000090: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................

I compared it with one of the later superblocks. I was lucky and had only lost the last one.

3. Get a temporary root directory onto the drive. For my particular case, I copied the start of an identically sized drive to the damaged drive so that I could get enough of a file system to work with.

dd if=/dev/sdd1 of=/dev/sdc1 count=2000 bs=4096

4. Scan the drive and figure out what directories you want to try to get back. I ended up writing a small Python program to scan the drive directly, finding the subdirectories that I wanted.

You’re looking for directories that are as high level as possible — in my case, I had a directory to_archive that contained all the things that I hadn’t redundantly backed up. It was at block 22839809.

5. Given a block number, it’s possible to convert it into an inode… but it’s much easier to just go to the directory in question and take a look at the inode for it.

dd if=/dev/sdc1 skip=22839810 count=1 bs=4096 | hex | head -n 1

The first four bytes are the inode. Copy them down.

5. I found the temporary root directory that I copied from the other drive.

python scan_directory.py /dev/sdc1
block # 1539
['.', '..', 'lost+found', 'var', 'etc', 'media', 'cdrom', 'bin', 'boot', 'dev', 'home', 'lib', 'mnt', 'opt', 'proc', 'root', 'sbin', 'srv', 'sys', 'tmp', 'usr', 'initrd.img', 'vmlinuz', 'initrd.img.old', 'vmlinuz.old']

This directory is mostly pointing into garbage, but we just want to point one directory to our stuff.

6. Now we want to point the inode of a directory to the one that contains our files.

dd if=/dev/sdc1 of=root_directory.bin count=1 bs=4096 skip=1539

The root directory looks something like this.

0x00000000: 02 00 00 00 0c 00 01 02 - 2e 00 00 00 02 00 00 00 ................
0x00000010: 0c 00 02 02 2e 2e 00 00 - 0b 00 00 00 14 00 0a 02 ................
0x00000020: 6c 6f 73 74 2b 66 6f 75 - 6e 64 00 00 01 e0 0a 00 lost+found......
0x00000030: 0c 00 03 02 76 61 72 00 - 01 40 17 00 0c 00 03 02 ....var..@......
0x00000040: 65 74 63 00 01 e0 0c 00 - 10 00 05 02 6d 65 64 69 etc.........medi
0x00000050: 61 00 00 00 0c 00 00 00 - 10 00 05 07 63 64 72 6f a...........cdro
0x00000060: 6d 00 00 00 01 00 12 00 - 0c 00 03 02 62 69 6e 00 m...........bin.
0x00000070: 01 60 20 00 0c 00 04 02 - 62 6f 6f 74 01 40 13 00 .` .....boot.@..
0x00000080: 0c 00 03 02 64 65 76 00 - 01 20 1b 00 0c 00 04 02 ....dev.. ......
0x00000090: 68 6f 6d 65 01 40 03 00 - 0c 00 03 02 6c 69 62 00 home.@......lib.

The name of a file is preceded by a a) one byte type, b) a one byte name length, c) two bytes of record length, and d) four bytes of an inode. In this case, we’re going to change lost+found from inode 0x0b to the one that we got in step 4. Easiest way is to use dd and tweak:

tweak root_directory.bin
... edit it ...
dd if=root_directory.bin of=/dev/sdc1 count=1 bs=4096 seek=1539

7. Now, mount it and cross your fingers.

mount -o ro /dev/sdc1 /mnt/external

In my case, I was lucky — I was able to get everything back in the directories that I cared about.

brool » ext3

Recovering A Trashed Ext3 System