raven ioctl

Backing up and recovering a Debian installation on a machine via USBstick and system image on an XP workstation

ioctl.org : unix bits and pieces : debian bare metal recovery

Debian boot device creation from scratch and bare metal recovery

This is just a brief summary from first principles of what should go onto a bootable USB key in order to enable a bare metal recovery of a Debian media PC. I wrote it because I couldn't find anything that actually fit the bill elsewhere, although various bits and pieces helped to put the puzzle together.

The problem statement

Again, this was written with James in mind: a Debian newbie (whose ambitions are limitless). His setup - several PCs on a LAN, most of which run Windows of some flavour or another. He has a single small "media PC" onto which we installed Debian. The media PC is USB- and network-bootable, but has no floppy drive or CD. Given the availability of other Windows boxes on the LAN, he wanted a simple way to back up his machine contents to one of those boxes (as a staging point from whence harder copy, such as DVD, could be produced). Additionally, he wanted a simple mechanism for a DR situation that would let him recover his media PC. The DR situation in question assumes that the other machines on the LAN are intact and have been recovered, but that he's looking at a blank or otherwise trashed media PC.

An outline of the solution

The solution falls into two pieces:

Backing up the system

The first step is to produce a backup of the system to restore later. This should include all the information required to recreate the system, assuming a starting point of an unpartitioned disk.

Locating the filesystems to dump

The mount command can be used to locate the filesystems that we want to preserve. We only back up non-transient, real data - no pseudo filesystems.

#
mount
show the mount points
/dev/hda1 on / type ext3 (rw)
this is the only filesystem to dump
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /dev type tmpfs (rw,size=10M,mode=0755)
usbfs on /proc/bus/usb type usbfs (rw)

Backup: smbmount & dump

The backup process is a simple one. Use smbmount to make the remote Windows filesystem available. Use the dump command to make a full backup of the filesystem. We compress this and split it into manageable chunks at the same time. (Not every Windows filesystem likes huge files; we split at around the 1GB mark.) Repeat for every real, on-disk filesystem. We don't need to back up /proc, /dev and the like.

#
smbclient -NL 192.168.1.1
locate the available share
Anonymous login successful
Domain=[IOCTL] OS=[Unix] Server=[Samba 2.2.12]
Sharename Type Comment
--------- ---- -------
share Disk
this is the sharepoint we'll use
IPC$ IPC IPC Service (tribble)
ADMIN$ Disk IPC Service (tribble)
Anonymous login successful
Domain=[IOCTL] OS=[Unix] Server=[Samba 2.2.12]
Server Comment
------ -------
TRIBBLE tribble
Workgroup Master
--------- -------
IOCTL
#
smbmount //192.168.1.1/share /mnt -o guest
mount the remote share (as a guest)
Anonymous login successful

Depending on the details of how you've shared out the folder you're going to be using, you may have to supply credentials to smbclient and smbmount.

Anyway, once you've mounted the remote folder, you need to dump your filesystem(s) onto it. Some Windows versions don't support large files, and some smbfs implementations don't either (although I think if you're using a modern kernel against Windows XP you should be fine) so for safety's sake, we use split to chop the backup file into manageable chunks.

The package I use for actually performing the backup is "dump", which provides the BSD-style dump/restore commands.

#
apt-get install dump
if it's not already installed
Reading Package Lists... Done
Building Dependency Tree... Done
dump is already the newest version.
(it is in this case)
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#
dump -0af - / | bzip2 -9 | split -b1000m - /mnt/dump-root-20051013
repeat this command for each filesystem
DUMP: Date of this level 0 dump: Thu Oct 13 13:34:04 2005
DUMP: Dumping /dev/hda1 (/) to standard output
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 112 blocks.
I used a much smaller filesystem for this example
DUMP: Volume 1 started with block 1 at: Thu Oct 13 13:34:04 2005
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: Volume 1 completed at: Thu Oct 13 13:34:04 2005
DUMP: Volume 1 110 blocks (0.11MB)
DUMP: 110 blocks (0.11MB)
DUMP: finished in less than a second
...so it finished very quickly
DUMP: Date of this level 0 dump: Thu Oct 13 13:34:04 2005
DUMP: Date this dump completed: Thu Oct 13 13:34:04 2005
DUMP: Average transfer rate: 0 kB/s
DUMP: DUMP IS DONE

Real filesystem dump times will take longer. For example: in James' setup, with a single filesystem of about 2.7GB, it took around 80 minutes to write out a compressed dump that totalled about 1.3GB. That's around 33MB/s off the disk and through the compression filter and 16MB/s over the network and onto the Windows disk.

This process finishes with a small number of files on the remote share, each of which (apart from the last one, which may be smaller) is 1024000kB in size.

#
ls /mnt
dump-root-20051013aa dump-root-20051013ab

Backing up metadata

There is usually a small amount of additional metadata that is required in order to perform a bare-metal recovery. In this case, the actual disk layout (the details of the various fdisk partitions) will need to be known if they are needed to be recreated. We can use several tools to get this information - we picked parted in this case.

#
parted /dev/hda print > /mnt/parted-20051013
save the partition layout for recovery use
#
cat /tmp/parted-20051013
(we'll have a peek at what's saved)
Disk geometry for /dev/hda: 0.000-38166.679 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 36239.765 primary ext3 boot
2 36239.796 38166.679 extended lba
5 36239.796 38166.679 logical linux-swap
Information: Don't forget to update /etc/fstab, if necessary.
#
dumpe2fs -h /dev/hda1 > /mnt/dumpfs-root-20051013
this is only required if we want to record non-default options to mkfs
#
umount /mnt
we're done with this for now

If the filesystems themselves have any special, non-default parameters set, then that information will also be required. Example parameters are things like the number of bytes per inode. You'd need to know this if you wanted to recreate the filesystems using the same parameters. In our case, we were faced with a single large ext3 root filesystem (with default parameters) and a swap partition - pretty straightforward.

Preparing for bare-metal recovery

The basic plan for the bare-metal recovery can be stated as follows: we prepare a bootable USB key that contains an absolutely minimal functioning operating system, together with sufficient tools to recover the system from the information we've got stashed on the (network-available) Windows share.

The tasks we'll face, given a working USB key, are as follows:

Making a bootable USB key

Having used SYSLINUX before to actually install Debian onto the Mini PC from a USB key, we knew that this process could be made to work (in principle). There is a version of syslinux available for Debian, so we should be able to prepare the boot stick from within the Linux environment.

SYSLINUX, vmlinuz and initrd

The first step was to use syslinux to prepare the boot device. Inserting the keyfob into the USB port, we were able to mount it (it was already formatted and partitioned) and ready it.

#
ls /dev/sd*
/dev/sda /dev/sda1
#
mount /dev/sda1 /mnt
#
ls /mnt
#
apt-get install syslinux
Reading Package Lists... Done
Building Dependency Tree... Done
syslinux is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#
syslinux /dev/sda1
#
ls /mnt
ldlinux.sys

There are three additional things that are needed on the boot device in order for it to work. These are: a configuration file for syslinux, a kernel image and an initial filesystem image.

#
cat > /mnt/syslinux.cfg
default linux
label linux
kernel vmlinuz
append initrd=initrd ramdisk_size=30720 root=/dev/rd/0 devfs=mount rw --
hit control-D here
#
umount /mnt
we're done with this for now

We will supply a the referenced files, vmlinuz and initrd, shortly. To understand what these parameters do, you need to understand the linux boot process.

An outline of the boot process

Assuming the BIOS is capable of booting from USB, the boot sector of the key (which syslinux writes) will load and run the loader, ldlinux.sys. This, in turn, uses BIOS I/O calls to load the kernel image (vmlinuz) and, if specified, an initial filesystem image (initrd). This filesystem image resides in RAM, and is used to bootstrap the system in normal operation. For our purposes, however, we will produce a custom initrd that contains just the bare essentials required to perform our recovery.

If initrd is specified, then after the kernel is running, the first process it launches is /linuxrc in the RAM filesystem. This can be anything executable: a binary file or a shell script (assuming the shell interpreter and all its requirements are also present in the ramdisk image). Once the /linuxrc process exits, the kernel expects a "real" root filesystem to be in place and will launch /sbin/init as per usual.

Details of the boot process can be found in the kernel source. If you want to have a look...

#
apt-get source kernel-tree-2.6.8
Reading Package Lists... Done
Building Dependency Tree... Done
Need to get 44.8MB of source archives.
Get:1 http://example.org sarge/main kernel-source-2.6.8 2.6.8-16 (dsc) [989B]
Get:2 http://example.org sarge/main kernel-source-2.6.8 2.6.8-16 (tar) [43.9MB]
Get:3 http://example.org sarge/main kernel-source-2.6.8 2.6.8-16 (diff) [912kB]
Fetched 44.8MB in 8s (5045kB/s)
dpkg-source: extracting kernel-source-2.6.8 in kernel-source-2.6.8-2.6.8
#
cd /usr/src/kernel-source-2.6.8-2.6.8/Documentation
#
more kernel-parameters.txt \
>
i386/boot.txt \
>
initrd.txt \
>
filesystems/devfs/boot-options
these are what I read to get this working

Types of initrd

There appear to commonly be two formats for the initrd image that are used by various Linux loaders and boot systems. In no particular order, these are: a gzipped ext2 filesystem image; a special compressed boot filesystem called cramfs. The latter is not an updatable (writable) format once mounted.

Assuming that you've got a directory hierarchy in initrd-root/ and you want to turn its contents into a gzipped ext2, here's how you go about it. Note, I'm skipping ahead slightly: once the process for creating an initrd image is understood, I'll come back to the question of what contents it'll need, exactly.

#
du -sxk initrd-root
get a rough idea of how big the filesystem will have to be
8048 initrd-root
around 8MB
#
dd if=/dev/zero of=initrd bs=8k count=2048
preallocate 16MB to hold the filesystem
2048+0 records in
2048+0 records out
16777216 bytes transferred in 0.079912 seconds (209946266 bytes/sec)
#
losetup -f
identify the next available loopback device
/dev/loop0
#
losetup -f initrd
set up a pseudo-device backed by the file contents
#
mkfs -t ext2 /dev/loop0
create a filesystem...
mke2fs 1.37 (21-Mar-2005)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
4096 inodes, 16384 blocks
819 blocks (5.00%) reserved for the super user
First data block=1
2 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
8193
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
#
mount /dev/loop0 /mnt
#
cp -r initrd-root /mnt
...populate it...
#
umount /mnt
#
losetup -d /dev/loop0
...and tear down the psuedo-device
#
gzip -9 initrd
finish by compressing the filesystem image
#
ls -l initrd.gz
-rw-r--r-- 1 root root 3330231 2005-10-13 16:00 initrd.gz

If you follow the route above, you'll need to set initrd.gz as the value of the initrd parameter in syslinux.cfg.

Alternatively, you can create a cramfs image in a rather more straightforward fashion:

#
mkcramfs initrd-root initrd
Directory data: 4392 bytes
Everything: 3656 kilobytes
Super block: 76 bytes
CRC: 1d316bab
warning: gids truncated to 8 bits (this may be a security concern)
#
ls -l initrd
-rw-r--r-- 1 root root 3743744 2005-10-13 16:20 initrd

Which filesystem type should you use? Well, cramfs looks simpler to manage, although it has a limitation (it's not able to be written to/ updated once created) which we'll see shortly: for the purposes of experimentation, that limitation might get in the way.

The decision as to filesystem type is actually forced upon you by a chicken-and-egg situation: modern Linux kernels tend to use loadable modules for much of their functionality, filesystems included. Some functionality is built directly into the kernel, some as a loadable module. But at the point where the kernel has just been bootstrapped by ldlinux.sys, no loadable kernel modules are available: they're in the ramdisk filesystem! The kernel must have a stand-alone capability of understanding the ramdisk image, which is to say, that the filesystem type in question (ext2 or cramfs) must have been built into the kernel.

In the case of James' Debian installation, the kernel has in-built cramfs support, but ext2 is available only via a module. Rather than build a custom kernel for the USB stick, our plan was to use a copy of the working kernel (that is, whatever Debian booted into normally) as the basis of our recovery image. So we went with cramfs.

Kernel parameters

The other kernel parameters that ldlinux.sys supplies to the bootstrapped kernel are explained in the kernel source Documentation directory. The important one is devfs=mount. Although devfs is now deprecated, this pseudo-filesystem (which is mounted automatically over /dev by this parameter) is very useful: device nodes are created automatically in it as the kernel detects devices. As such, it is a great tool for telling when you've loaded all the kernel modules you need (the driver will create the device node) - as well as not having to worry about what major and minor numbers are associated with a particular driver. The alternative to devfs is to manually create the device nodes we'll require in the initrd-root directory beforehand. This is a possibility, but a clumsy one and one that we'd rather avoid if possible.

First stab at a boot image

Having understood how to create an initrd image, given a pre-prepared directory hierarchy, we now address the question of what that hierarchy should contain. The initial list I used follows.

The first question is what kernel modules we should make available to our boot environment. Here, we cheated: since the plan is to use a kernel that we know works (because it is the kernel that runs the Debian installation normally), we just used lsmod to get a list of modules that are loaded after normal day-to-day operation. To that list, we add smbfs for smbmount support.

#
mkdir initrd-root
these examples use zsh, which has /bin/sh-compatible syntax
#
copy_file() {
this first function copies a named file into our initrd-root hierarchy
>
local file="$1"
>
if [ ! -e "$workdir/$file" ]
>
then
>
echo " Copying file $file"
>
mkdir -p "$workdir/$(dirname "$file")"
>
cp -p "$file" "$workdir/$file"
>
fi
>
}
#
copy_module_aux() {
this one locates a module by name and uses copy_file to copy it
>
local module="$1"
>
find "$moduledir" -name "$module".ko | (
>
while read file
>
do
>
copy_file "$file"
>
done
>
)
>
}
#
copy_module() {
the module foo_bar may be provided...
>
local module="$1"
>
echo "Copying module $module"
>
copy_module_aux "$module"
... by foo_bar.ko ...
>
copy_module_aux "$(echo "$module" | tr _ -)"
... or by foo-bar.ko
>
}
#
workdir=`pwd`/initrd-root
supply the working directory for our functions
#
moduledir=/lib/modules/2.6.8-2-686
and the directory to copy modules from
#
copy_module smbfs
ensure our initrd will have the smbfs module
Copying module smbfs
Copying file /lib/modules/2.6.8-2-686/kernel/fs/smbfs/smbfs.ko
#
lsmod | tail +2 | awk '{print $1}' |
get the names of currently loaded modules...
>
while read mod; do copy_module "$mod"; done
... and copy each one
Copying module isofs
Copying file /lib/modules/2.6.8-2-686/kernel/fs/isofs/isofs.ko
(output elided)
Copying module e100
here are our network drivers
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/net/e100.ko
Copying module mii
lsmod shows this is required too
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/net/mii.ko
(output elided)
Copying module ext3
support for our filesystem of choice
Copying file /lib/modules/2.6.8-2-686/kernel/fs/ext3/ext3.ko
Copying module jbd
Copying file /lib/modules/2.6.8-2-686/kernel/fs/jbd/jbd.ko
(output elided)
Copying module ide_generic
driver support for our IDE disk
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/ide/ide-generic.ko
Copying module ide_disk
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/ide/ide-disk.ko
Copying module ide_core
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/ide/ide-core.ko
(output elided)
Copying module cfbfillrect
eventually it finishes
Copying file /lib/modules/2.6.8-2-686/kernel/drivers/video/cfbfillrect.ko
#
du -sxk initrd-root
get some idea of how big our initrd is so far
3472 initrd-root

As shown above, we begin to build a root filesystem hierachy for our initrd in initrd-root. Many of the modules that a normally-running kernel requires will not be needed for recovery operation; however, this tactic is a quick way to grab pretty much everything that will be required.

A look at the output of lsmod is instructive. In particular, jot down likely drivers (as I've highlighted above) plus any modules that are depended upon by those drivers. For example, the IDE stack comprises ide_core, ide_disk and ide_generic. You may have machine-specific drivers for your chipset too; in the case of the MiniPC, we needed some VIA chipset support.

We're building a recovery filesystem, so we'll need some empty directories in there to mount filesystems over.

#
mkdir initrd-root/tmp
#
mkdir -p initrd-root/mnt/smb
we'll mount the recovery Windows share here
#
mkdir initrd-root/mnt/hd
we mount our hard disk root filesystem here
#
mkdir initrd-root/proc
many basic utilities require /proc in order to operate
#
mkdir initrd-root/dev
the devfs filesystem will mount here

Remember, this is for bare-metal recovery. When we recover the system, there is no guarantee that the hard drive in it will be partitioned or have the correct filesystems on it. The recovery shell will run out of a ramdisk; so we need the utilites present to partition the hard drive, make a filesystem on it, mount the filesystem, mount the remote share, and run the recovery. Additionally, we'll need a shell, and the basic file utilities that you probably take for granted.

We use busybox as a convenient way to supply many of the standard commands. For a shell, we'll use ash. Throw into the mix the fsck tools, mkfs, smbmount, bunzip2 and restore to perform the recovery. What's missing? We use parted for hard disk partitioning. One non-obvious requirement is insmod, a low-level tool for loading modules into a running kernel. Without that, our effort locating kernel modules will have been for naught!

Many of these utilities use dynamically-loaded libraries: that is, there are a handful of hidden dependencies. The ldd command can find these library dependencies. Note: some libraries actually depend on additional libraries, so we use ldd on those too. The following small script will take care of this (this continues the previous session).

#
apt-get install busybox dash bzip2 smbfs dump
these are the tools we want on the recovery initrd
Reading Package Lists... Done
Building Dependency Tree... Done
busybox is already the newest version.
I installed these earlier
dash is already the newest version.
bzip2 is already the newest version.
smbfs is already the newest version.
dump is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
#
cat > /tmp/exes
make an initial list of executables to copy
/bin/busybox
/bin/dash
/sbin/pivot_root
I'll explain this one shortly
/sbin/insmod
/sbin/parted
/sbin/mkfs
/sbin/mkfs.ext2
/sbin/fsck
/sbin/fsck.ext2
/usr/bin/smbmount
/usr/bin/smbmnt
this is required by smbmount: a dependency I discovered the hard way
/usr/bin/bunzip2
/sbin/restore
hit control-D here
#
while read exe
>
do
>
if [ "x$exe" = "x" ]; then continue; fi
>
if [ -e "$workdir/$exe" ]
>
then
>
echo "Executable $exe (already dealt with)"
>
continue
>
fi
>
if [ ! -e "$exe" ]
>
then
>
echo " WARNING $exe does not exist to copy" 1>&2
>
continue
>
fi
>
echo " Executable $exe"
>
copy_file "$exe"
>
>
ldd "$workdir/$exe" | awk '$2 == "=>" { print $3}' >> /tmp/exes
for each executable, we add any dynamic dependencies
>
done < /tmp/exes
Executable /bin/busybox
Copying file /bin/busybox
Executable /bin/dash
Copying file /bin/dash
Executable /sbin/pivot_root
Copying file /sbin/pivot_root
Executable /sbin/insmod
Copying file /sbin/insmod
Executable /sbin/parted
Copying file /sbin/parted
Executable /sbin/mkfs
Copying file /sbin/mkfs
Executable /sbin/mkfs.ext2
Copying file /sbin/mkfs.ext2
Executable /sbin/fsck
Copying file /sbin/fsck
Executable /sbin/fsck.ext2
Copying file /sbin/fsck.ext2
Executable /usr/bin/smbmount
Copying file /usr/bin/smbmount
Executable /usr/bin/smbmnt
Copying file /usr/bin/smbmnt
Executable /usr/bin/bunzip2
Copying file /usr/bin/bunzip2
Executable /sbin/restore
Copying file /sbin/restore
Executable /lib/tls/libc.so.6
Copying file /lib/tls/libc.so.6
Executable /lib/ld-linux.so.2
Copying file /lib/ld-linux.so.2
Executable /lib/tls/libc.so.6 (already dealt with)
... and so on
#
du -sxk initrd-root
how large is our (unpacked) initrd now?
9472 initrd-root

Finally, busybox can modify its behaviour depending on how it is invoked (a switch on the value of argv[0]). Similarly, we'd like to make /bin/sh a sysnonym for dash. Additionally, fsck.ext2 and fsck.ext3 are the same program. So we add some symbolic links to our initrd. In the script below, the targets of these links are absolute, not relative. That's ok; in use, the contents of initrd-root will be mounted at the root directory.

We finish our symlinking by making /linuxrc a pointer to /bin/dash; this should drop us into a shell after the kernel finishes its bootstrap process.

#
ln -s /bin/dash initrd-root/bin/sh
#
ls -l initrd-root/bin/sh
lrwxrwxrwx 1 root root 9 2005-10-17 14:01 initrd-root/bin/sh -> /bin/dash
this is what it looks like
#
ln -s /sbin/mkfs.ext2 initrd-root/sbin/mkfs.ext3
#
ln -s /sbin/fsck.ext2 initrd-root/sbin/fsck.ext3
#
ln -s /bin/dash initrd-root/linuxrc
#
ln -s /bin/busybox initrd-root/bin/[
#
ln -s /bin/busybox initrd-root/bin/ar
#
ln -s /bin/busybox initrd-root/bin/cat
#
ln -s /bin/busybox initrd-root/bin/chgrp
... and so on for each command that busybox provides
#
ln -s /bin/busybox initrd-root/bin/whoami
#
ln -s /bin/busybox initrd-root/bin/zcat

(Note: there are a set of scripts provided at the end of this document that take a lot of the drudgery out of the processes I describe here.)

Finishing and testing the boot key

It's time to finish up and test the boot key. We do this by remounting the keyfob filesystem and copying our kernel and initrd images back onto it.

#
mount /dev/sda1 /mnt
remount our keyfob
#
cp /boot/vmlinuz-2.6.8-2-686 /mnt/vmlinuz
copy the kernel to the boot device
#
mkcramfs initrd-root initrd
update the compressed initrd image
Directory data: 3884 bytes
Everything: 4260 kilobytes
Super block: 76 bytes
CRC: 398b9151
warning: gids truncated to 8 bits (this may be a security concern)
#
ls -l initrd
how large is it these days?
-rw-r--r-- 1 root root 4362240 2005-10-17 14:22 initrd
#
cp initrd /mnt
copy the initrd to the boot device
#
umount /mnt
unmount the keyfob

Crunch time: time to try the recovery device! Reboot your system with the USB key inserted. You may have to fiddle with the BIOS settings to ensure that the USB key appears before your (currently functional and bootable) hard drive as a boot device. If all goes well, you'll see SYSLINUX load and start your kernel; you should see some starting kernel device probe messages, followed by a shell prompt: that's linuxrc running, which in this case is the /bin/dash interpreter.

Immediate pitfalls

There are some pitfalls to be aware of with our rudimentary environment. Some of these are more problematic than others. That's ok; this first session is simply to test the water.

cramfs / is not writable

There are a number of occasions where it is convenient to be able to write to the ramdisk image. This is not possible with a cramfs root filesystem. There is a way to replace the root filesystem with a writable memory-based filesystem, once the initial boot is complete: that is documented shortly.

/proc is a fundamental requirement

Many very basic utilities, such as ifconfig and ps, require a working /proc filesystem in order to function at all. The first time I built the initrd image, I neglected to include an empty directory to mount /proc over; because cramfs is not writable, I was unable to run mkdir /proc in order to create the mount point. (Actually, it's possible to work around such a problem, but it was simpler to reboot into the operating system proper in order to update the initrd image on the USB stick.)

Missing utilities

After my second attempt at an initrd image (this one with a /proc) the first missing utility I discovered was insmod. (You'll note I've skipped this learning step in the scripted session above.) Without this, it's impossible to load any kernel modules into the running system.

Other non-obvious missing dependencies include some additional support required for smbmount. (In fact, to speed things along, I've added the smbmnt executable to the list above - when I initially did this work, I found out that it was required because smbmount complained that it couldn't be found.) Not to worry, we can jot down a list of missing things and rebuild our initrd. Once the boot disk is proved in principle - that is, the hardware drivers are demonstrated to be working - it's possible to speed development up by using a virtual machine to debug some of the higher-level utilities. I'll outline that process later.

Punctuation is all over the place

I use, and am used to, the GB layout for keyboards. By default, the Debian kernel uses some other keyboard layout (US?) - consequently, handy keys like the quotes, the pipe, etc, aren't necessarily where you think they might be. Mostly, I can get by with a little trial-and-error.

Lack of job control

I must confess that I found this out the hard way. The initial environment is very basic; amongst other things, it has no support for job control. The most marked consequence of this is that the interrupt key sequence, control-C, will not work. I discovered this after I managed to get a network interface up: ping in its most basic invocation entered an infinite loop, necessitating a reboot after I'd run it. Oops!

Making a basic working environment

So, we've booted into the recovery environment. We sit, staring at a root shell prompt. What're the next steps? We'll address some of the issues highlighted above, then move on to getting our recovery going.

Our first task is to get our basic environment sorted out: to get the prerequisites working before we tackle the network and disk subsystems. The initial step is to get a writable root filesystem working. (We won't need this immediately, but it makes life much simpler in the long run; and this issue is technically simpler if we address it first.) The process: we do this by creating a new, writable, virtual-memory-backed filesystem (tmpfs) - this has the added advantage that once swap space is enabled, its contents can be paged out to disk. Having made and mounted the filesystem, we carefully copy the contents of the existing root filesystem into it. Finally, we invoke pivot_root, which essentially swaps the relative positions of the two filesystems: the new filesystem becomes the new root, and the old filesystem winds up mounted in a subdirectory under it.

#
mount -t tmpfs - /tmp
/tmp will hold our new root filesystem for the moment
#
cp -a /bin /sbin /usr /lib /mnt /tmp
copy the (normal) directory contents from the root filesystem
#
mkdir /tmp/dev /tmp/devfs /tmp/proc /tmp/old-root
create new mount-points under the to-be-root directory
#
cd /tmp
#
mount -t devfs - devfs
we mount a second devfs under what will become /devfs
#
cd /tmp/dev
#
ln -s ../devfs/* .
this trick will give us an updatable /dev directory with devices in it
#
cd /tmp
#
pivot_root . old-root
what was /tmp now becomes /
what was / now becomes /old-root
#
exec chroot . /bin/sh
respawn the shell process (this is part of the pivot_root step)

The next step is to get a working /proc.

remember, these commands are issued to the recovery shell
#
ifconfig -a
what network devices are available?
Warning: cannot open /proc/net/dev (No such file or directory). Limited output.
ifconfig needs /proc to work fully
#
mount -t proc - /proc
make /proc available ...
#
ifconfig -a
... and try again
lo Link encap:Local Loopback
loopback support is compiled into the kernel
LOOPBACK MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 iB) TX bytes:0 (0.0 iB)

At that point we can unmount the old root (and the devfs instance it contains).

#
umount /old-root/dev
umount: /old-root/dev: Device or resource busy
because our shell has the old /dev/console open
#
exec /bin/sh < /devfs/console > /devfs/console 2>&1
close the old devices and start a shell connected to the console under /devfs
#
umount /old-root/dev
now this works
#
umount /old-root
ditch the old cramfs

Getting networking working

The next milestone is to connect our newly-booted machine to the LAN. In our case, the machine normally lived on a private subnet, 192.168.1.0/24. It would usually have an IP address allocated out of a DHCP pool. For the purposes of recovery, we instead decided to manually allocate an IP address to the machine (one that didn't coincide with the DHCP pool). Before we can configure eth0, however, we actually have to get the network device recognised by the kernel. To do this, we need to load the appropriate modules to support the card. These will vary depending on hardware; in our case, we were able to pull the likely driver suspects out of the lsmod output from a normal session with the machine booted into its full operating system.

#
insmod /lib/modules/2.6.8-2-686/kernel/drivers/net/mii.ko
first driver
these paths are a little unwieldy!
#
loadmod() {
define a function to make life simpler
>
insmod $(find /lib/modules -name "$1")
>
}
#
loadmod crc32.ko
#
loadmod via-rhine.ko
the MiniPC's chipset is VIA
probe messages elided
#
ifconfig -a
eth0 Link encap:Ethernet HWaddr aa:bb:cc:dd:ee:ff
our interface is now recognised
BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 iB) TX bytes:0 (0.0 iB)
Interrupt:9 Base address:0xc100
lo Link encap:Local Loopback
and here's an (unconfigured) loopback
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 iB) TX bytes:0 (0.0 iB)

Now that the kernel knows about the new network device, we can configure it with an IP address and try to ping our recovery machine (at 192.168.1.1). We also configure the loopback interface at the same time. Note, we give ping a count to prevent it running an infinite loop - remember, there is no job control available in this shell.

#
ifconfig lo 127.0.0.1 netmask 255.0.0.0
#
ifconfig eth0 192.168.1.99 netmask 255.255.255.0
out of the way of our DHCP pool
#
ping -c 3 127.0.0.1
check the loopback works
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.033 ms
--- 127.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.033/0.033/0.034/0.004 ms
#
ping -c 3 192.168.1.1
the IP of our Windows server
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.250 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.191 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.206 ms
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms

Success! This represents a major milestone: if our recovery boot device is able to see the recovery host on the LAN, we're close to completion.

Note that we only needed to load kernel support for our particular network card. The stock Debian kernel we used has the loopback device built in, as well as the rest of the IPv4 framework pieces that we'll require.

Getting smbmount working

With basic TCP/IP networking configured, the next step is to get smbmount working to access the share on the recovery PC. This proves to be trickier: SMB support comes with a whole raft of issues. We are satisfied with a less-than-complete solution providing it suffices to let us grab our dump images for the purposes of running a system recovery. If your native language is Chinese, you might need to work harder at the SMB codepage support.

We jump right in, and try to mount the remote filesystem. Here are the steps to follow:

#
loadmod smbfs.ko
smbmount can't load the module on its own in this cut-down environment
#
smbmount //192.168.1.1/share /mnt/smb -o guest,ro
params.c:OpenConfFile() - Unable to open configuration file "/etc/samba/smb.conf":
No such file or directory
init_iconv: Conversion from UTF-16LE to CP850 not supported
init_iconv: Attempting to replace with conversion from UTF-16LE to ASCII
init_iconv: Conversion from UTF-8 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from ANSI_X3.4-1968 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to UTF-16LE not supported
init_iconv: Attempting to replace with conversion from ASCII to UTF-16LE
init_iconv: Conversion from CP850 to UTF-8 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to ANSI_X3.4-1968 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to UTF8 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from UTF8 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
creating lame upcase table
creating lame lowcase table
init_iconv: Conversion from UTF-16LE to CP850 not supported
init_iconv: Attempting to replace with conversion from UTF-16LE to ASCII
init_iconv: Conversion from UTF-8 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from ANSI_X3.4-1968 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to UTF-16LE not supported
init_iconv: Attempting to replace with conversion from ASCII to UTF-16LE
init_iconv: Conversion from CP850 to UTF-8 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to ANSI_X3.4-1968 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from CP850 to UTF8 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
init_iconv: Conversion from UTF8 to CP850 not supported
init_iconv: Attempting to replace with conversion from ASCII to ASCII
Can't load /etc/samba/smb.conf - run testparm to debug it
#
mount
rootfs on / type rootfs (rw)
devfs on /dev type devfs (rw)
- on / type tmpfs (rw)
- on /devfs type devfs (rw)
- on /proc type proc (rw,nodiratime)
//192.168.1.1/share on /mnt/smb type smbfs (ro,nodiratime,nosuid,nodev,file_mode=0755,dir_mode=0755)
despite these errors, the mount succeeds
#
ls /mnt/smb
dumpfs-root-20051013 dump-root-20051013ab
here are the files we made earlier
dump-root-20051013aa parted-20051013

We ignore the code page translation errors and the lack of /etc/samba/smb.conf in the rest of this write-up. If your Windows host is more demanding, you may have to supply additional dependencies in your recovery environment. Such considerations are beyond the scope of this document.

In other words, this stage represents a "good enough" state of affairs. We're not looking to use this facility for day-to-day work - our motivation is simply to create a recovery environment, no more.

I've also arranged this write-up to avoid many of the errors I encountered whilst developing this process. For instance, smbmnt tries to write to /etc - an issue which I skirted by beginning the session with the replacement of the root filesystem with a writeable one.

Getting the hard drive working

Now that we can see the remote dump contents, the next step is to get the kernel to talk to the local hard drive. The modules we needed to get this working on the MiniPC are shown below; your details may vary.

#
loadmod ide-core.ko
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 3MHz system has speed for PIO modes; override with idebus=xx
#
loadmod ide-generic.ko
probe messages elided
#
loadmod ide-disk.ko
probe messages elided
#
loadmod via82cxxx.ko
the VIA chipset support
probe messages elided
#
ln -s ../devfs/ide/host0/bus0/target0/lun0/disc /dev/hda
make device nodes for parted to operate on
#
ln -s ../devfs/ide/host0/bus0/target0/lun0/part1 /dev/hda1
make slice nodes for mkfs to operate on
#
ln -s ../devfs/ide/host0/bus0/target0/lun0/part2 /dev/hda2
make slice nodes for mkfs to operate on

Note that whether the target device nodes under /devfs will actually exist at this point depends greatly on the exact state of the hard drive in the PC. Nevertheless, we can create symbolic links at this stage: once the drive has been correctly partitioned, the partition devices will appear under /devfs and the symbolic links will resolve correctly.

We can now test parted. Let's have a look at the current state of the hard drive. Note: this is a non-destructive operation.

#
parted /dev/hda print
Disk geometry for /dev/hda: 0.000-38166.679 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 36239.765 primary ext3 boot
2 36239.796 38166.679 extended lba
5 36239.796 38166.679 logical linux-swap
Information: Don't forget to update /etc/fstab, if necessary.

An alternative way to select modules

I've used a pretty precise method here to select and load the minimal set of kernel modules that appeared to be necessary to perform these operations. If you're having problems, there is a more scatter-shot approach that may well work for you: take a running system, use lsmod to list the loaded modules. Reverse that list (since it shows the most-recently-loaded module at the top, with its dependencies beneath it) and use that order to load modules. That is:

on a normal running system:
#
lsmod | tail +2 | awk '{print NR " " $1}' |
>
sort -rn | awk '{print $2}' > /tmp/modules
copy /tmp/modules to /etc/modules in your initrd image
on the recovery system:
#
while read module
>
do
>
loadmod "$module".ko
>
loadmod "$( echo "$module".ko | tr _ - )"
>
done < /etc/modules
this will load every module you had on your running system

You may have some success with this approach if all else fails.

Testing without reboots

Thus far, our effort has taken place in a relatively safe environment: all testing has been non-destructive. From this juncture, that is no longer the case - parted, mkfs and restore will be used to write to the hard drive. We want to practice our disaster recovery process before a disaster happens; but we don't want to actually precipitate a disaster during the testing process.

Emulation: qemu

Our testing up to now has been to prove our concept: we can see the remote network share; we can see the hard drive. That testing has been done on real, live equipment because it is crucially dependent on having the right set of kernel drivers available. Having shown our plan to be feasible, we can now develop the rest of the process in a more controlled, less critical environment. For the purposes of testing, we use qemu, an emulator capable of reproducing a completely virtual PC.

The use of qemu has a double impact: firstly, the emulated hardware is not the same as that on the MiniPC. We will require a different set of kernel modules loaded to enable virtualised networking and the virtual hard drive. Secondly, qemu enables a much more rapid testing cycle: there are no more boot / write USB key / reboot cycles to deal with. Instead, we can prepare our boot image and launch qemu directly.

Booting qemu

Unfortunately, qemu is not yet capable of emulating a USB boot device. Instead, we fall back on an emulated boot CD image. This can be prepared using the isolinux facilities provided by the syslinux package and the standard ISO image creation tools.

#
mkdir iso-image
#
cp /usr/lib/syslinux/isolinux.bin iso-image
boot control for the ISO
#
cp /boot/vmlinuz-2.6.8-2-686 iso-image/vmlinuz
the kernel, just as for the USB device
#
cp initrd iso-image/initrd
and the initrd, identical to the USB device
#
cat > iso-image/isolinux.cfg
this is identical to the syslinux.cfg file
default linux
label linux
kernel vmlinuz
append initrd=initrd ramdisk_size=30720 root=/dev/rd/0 devfs=mount rw --
hit control-D here
#
mkisofs -o recovery.iso \
>
-b isolinux.bin -c boot.cat \
specify the boot parameters for the El Torito ISO
>
-no-emul-boot -boot-load-size 4 -boot-info-table \
>
iso-image
and the filesystem contents
#
ls -l recovery.iso
-rw-r--r-- 1 root root 6154240 2005-11-04 13:37 recovery.iso

Automation

In such a rapid development environment, the bottleneck becomes the speed of building our initrd images. Fortunately, the process is sufficiently mechanical that it can be captured with a set of scripts. Those scripts are available online at http://ioctl.org/unix/debian/debianboot.tar.gz and encapsulate the discussion in this document. The tarball contents are as follows:

Configuration

There are three small (28 lines in total, excluding comments) configuration files that live in the etc directory.

In addition, there is a skeleton of the initrd image:

Links, scripts, and so on can be created in this directory. Its contents will be used to form the basis of the constructed initrd image.

I've taken the time to write scripts that capture a lot of the processes outlined in this document. In particular, the current overlay contents include a set of startup scriptlets that will automatically run through the pivot_root sequence, populate the /dev directory and set up a handful of useful shell functions that automate some of the more laborious tasks outlined above.

You may wish to look through the contents of overlay: again, it is small - 77 symbolic links (most of these come from busybox) and around 10 files comprising roughly 60 lines of script. These are best read in the order they operate:

Executable scripts

Other directories that are created

A sample session

>
>
cd debianboot
>
sudo bin/install-prerequisites
>
cp etc/construct.conf.example etc/construct.conf
>
vi etc/construct.conf
or the editor of your choice
>
cp etc/modules.list.example etc/modules.list
>
vi etc/modules.list
uncomment the qemu support
>
cp etc/executables.list.example etc/executables.list
>
bin/prepare-qemu
ready the emulator environment
>
sudo apt-get install samba
if it's not already installed
>
sudo vi /etc/samba/smb.conf
add an entry similar to the following to your samba setup:
[qemu]
writable = no
path = /home/cmjg/debianboot/qemu/smb
public = yes
>
sudo /etc/init.d/samba restart
...and make samba pick up the change
with these prerequisites taken care of, the following is a typical cycle:
>
bin/construct-initrd
>
sudo bin/ready-usb-image
this script must be run as root
>
bin/ready-iso-image
... prepare a bootable ISO image for qemu
>
bin/qemu
launch an emulator and try it!
repeat this cycle until you're happy with the results
insert the memory stick to be prepared...
>
sudo bin/write-usb
you should now be holding a bootable recovery device!

Assuming your boot device works correctly under emulation, you can issue the following sequence of commands at the shell prompt in the qemu window: the following sequence assumes you've made the Samba configuration change given above.

all server output has been elided for clarity
#
qemu
set up devices and networking for an emulated environment
#
loadmod smbfs.ko
#
smbmount //10.0.2.4/qemu /mnt/smb -o guest,ro
the IP address is a Qemu-specific one: it connects to the samba server on your local machine
#
ls /mnt/smb
you should see the file contents at this point
you can close the Qemu window to finish

Using parted to recreate the partitioning scheme

We turn our attention to the hard disk. For the purposes of testing, we'll be practising these steps inside qemu, until we get them right. From now on, I'll also assume the scripts above are being used to prepare a working initrd. The initrd so produced includes the startup process outlined above, that also defines a shell command, qemu, which executes the following commands:

#
loadmod 8390.ko
loadmod is the convenience function shown previously
#
loadmod ne2k-pci.ko
for networking
#
ifconfig eth0 10.0.2.99 netmask 255.0.0.0
#
loadmod ide-core.ko
#
loadmod ide-disk.ko
#
loadmod ide-generic.ko
for IDE hard drive support

Boot up a fresh qemu instance, and run the qemu command. The first thing to do is to ascertain the current state of the hard disk: (note, /dev/hda is a symbolic link to the devfs device node that is created by overlay/scripts/12-devices)

#
qemu
server response elided
#
parted /dev/hda print
what is the current partition table?
Error: unable to open /devfs/ide/host0/bus0/target0/lun0/disc - unrecognised
disk label
Information: Don't forget to update /etc/fstab, if necessary.

In this case, the hard drive is completely blank, and needs a new partition table. Continuing the session:

#
parted /dev/hda mklabel msdos
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda print
Disk geometry for /devfs/ide/host0/bus0/target0/lun0/disc: 0.000-38166.679 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
Information: Don't forget to update /etc/fstab, if necessary.

Better, we now have a blank partition table. The next task is to recreate the partitions that we had originally, as saved by our backup process.

#
loadmod smbfs.ko
#
smbmount //10.0.2.4/qemu /mnt/smb -o guest,ro
#
cat /mnt/smb/parted-20050101
We want to recreate this
Disk geometry for /dev/hda: 0.000-38166.679 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 36239.765 primary ext3 boot
2 36239.796 38166.679 extended lba
5 36239.796 38166.679 logical linux-swap
#
parted /dev/hda mkpart primary ext3 0.031 36239.765
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda set 1 boot on
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda mkpart extended 36239.796 38166.679
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda set 2 lba on
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda mkpart logical 36239.796 38166.679
Information: Don't forget to update /etc/fstab, if necessary.
#
parted /dev/hda print
Disk geometry for /dev/hda: 0.000-38166.679 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 36240.380 primary boot
2 36239.796 38166.679 extended lba
5 36239.796 38166.679 logical
Information: Don't forget to update /etc/fstab, if necessary.
note, the exact partition details differ

The first thing to note is that the final partition layout after the parted session has completed differs from our saved layout. This is due to two features: firstly, the original disk was partitioned using a different partitioning program, which lined up the end of the disk slices differently; secondly, the qemu emulated disk may have different geometry to the original device.

Is this disparity important? If it is critical to your application, then some other partitioning utility, such as fdisk, may be better able to recreate your partitions. As it is, this difference is not critical. The dump format we use to save the filesystem contents is restored using filesystem (as opposed to block) operations and is consequently impervious to perturbations in the precise disk layout or partition size. All we need to ensure is that the resulting filesystem has sufficient space to hold our restored files.

Making the filesystem

The second thing to observe is that the "Filesystem" contents of the new partition table are blank. Although we have allocated space for the filesystem, we have not yet constructed it (it's "unformatted space", in Windows-speak). That's the next step.

Before performing this step, be careful. Although qemu is pretty clever in only allocating as much real disk space to hold its virtual disk image as it needs, the following commands will produce a qemu/hda file that's around 600MB in size. If you want to practice the process without generating such a large filesystem, you should edit the debianboot etc/construct.conf and allocate a smaller (say, 5GB) virtual disk. Note, you'll also need to trim the size of the partitions you create using parted if you do this. As a final reminder, you'll probably want to delete the qemu virual drive after you've finished practising; you can always recreate it, and you don't want it to bulk out the size of your backups.

#
touch /etc/mtab
the journal creation will fail without this
#
mkfs -t ext3 /dev/hda1
mke2fs 1.37 (21-Mar-2005)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
4643968 inodes, 9277529 blocks
463876 blocks (5.00%) reserved for the super user
First data block=0
284 block groups
32768 blocks per group, 32768 fragments per group
16352 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Writing inode tables: done
Creating journal and filesystem accounting information: done
This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Mounting the filesystem

With our target filesystem created, the next step is to mount the filesystem so that we can perform the restore. We use the /mnt/hd directory to mount the filesystem. This means that what appears as the /mnt/hd directory in our recovery environment will contain the root of our restored filesystem.

#
loadmod mbcache.ko
needed for ext3
#
loadmod jbd.ko
needed for ext3
#
loadmod ext3.ko
because mount can't find it automatically
#
mount -t ext3 /dev/hda1 /mnt/hd
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
#
ls -al /mnt/hd
drwxr-wr-w 3 0 0 4096 Nov 11 17:08 .
drwxr-wr-w 4 0 0 80 Nov 11 17:08 ..
dwrx------ 2 0 0 16384 Nov 11 17:08 lost+found

Success! The filesystem (which is currently empty) is mounted, ready to receive our restored data.

Note that the supplied debianboot.tar.gz contains a startup script (/scripts/40-filesystems) which automatically creates /etc/mtab and loads the kernel modules required to support the EXT3 filesystem; the commands required are given explicitly above, but are unnecessary if you base your recovery image on the one supplied.

Things are a little more complicated if your original file tree was comprised of multiple filesystems (that is, other filesystems were mounted under subdirectories of the root filesystem). This wasn't the case with James' small PC, so I won't cover it in detail; however the process is pretty straightforward. You need to recover the root filesystem first, or create by hand the mountpoints under your root filesystem, before restoring the other filesystem contents. It's possible to carefully juggle the order of these operations to migrate a system from one using multiple filesystems to a single filesystem, or vice-versa; however, that's beyond the scope of this document. If you understand the reasons why you might want to do it, you probably know enough to be able to perform the operation.

Restoring files

To review the current state of play: we have successfully mounted the remote Windows share that contains our compressed filesystem dump. We have repartitioned the hard drive and recreated our target filesystems. All that remains is to restore the dumped filesystem contents to the newly-created filesystem.

Interactive restore

Normally, the bare-metal recovery process would use a full automated recovery (which is shown below). However, you might wish to try to recover a subset of the filesystem instead. The restore utility supports an interactive recovery session which lets you navigate the available files and select those you wish to restore.

#
rm /dev/tty && ln -s /dev/console /dev/tty
restore needs a working /dev/tty for interactive mode
(the step above is normally performed by /scripts/12-devices)
#
cd /mnt/hd
because restore recovers files relative to the current directory
#
cat /mnt/smb/dump-root-20051013* |
join up the split recovery image
>
bunzip2 -cd |
and uncompress it
>
restore -iaf -
and perform an interactive recovery

Note that even this partial recovery will take some time. This is because the compressed filesystem dump is read as a stream (rather than being randomly accessible). The amount of time required should be roughly the same as the time taken for the full dump.

Noninteractive restore

Restore can also be used in noninteractive mode: for most base-metal recovery scenarios, an automatic full restore of all files is usually the desired operation.

#
cd /mnt/hd
#
cat /mnt/smb/dump-root-20051013* |
join up the split recovery image
>
bunzip2 -cd |
and uncompress it
>
restore -rf -
and perform a full recovery
#
rm restoresymtable
this file is left behind by restore

The full recovery will take some time: around the same amount of time as a full dump took, under normal operation.

Making the disk bootable: grub

We don't have to make a copy of grub because it'll be available on the restored system.

Finishing touches

Save USB stick contents and a windows version of SYSLINUX to the DVD with the rest of your recovery materials. Practice this process. If you have a DR process, it's no good unless you're making regular backups.

Further reading

Links to: Linux documentation syslinux busybox qemu find a good dump/restore doc.