Custom Debian live ISO

By vanaalten on donderdag 14 oktober 2021 12:45 - Comments (4)
Category: -, Views: 2.885

Edit: updated content based on feedback from eXcyle and sfranken - thanks!

foreword
Another blogger adding a blog to TweakBlogs? Yes. But, don't expect a series of blogs from me, probably only this single blog because: I think I have something useful to post, but I don't like to make an effort to make it visually appealing (so, no custom CSS, no pictures, no background, no spell checking...) and probably no inspiration for any new blogs. So, just this one. Let's get started...


Introduction
...so a looooong time ago, I decided it would be a good idea to have an off-site backup for the most critical data on my system.

System: a simple headless server running Debian Linux, doing mail, downloads, fileserving, that kind of stuff. I've actually got two identical systems and every night, the 'always on' system is mirrored to the 'always off' system. If the 'always on' system breaks down, I can easily switch to the backup system.

But, just for the fun of it, and because it sounded like a good idea, I've also set up an off-site backup for the most critical data. Using duplicity, I'm backing up to Backblaze B2. Originally to Amazon S3, but Backblaze was significantly cheaper. Data is encrypted before being transmitted to Backblaze, so I'm not worried about my personal data being 'out there'.

One thing I've neglected to do for a long time: storing the keys/passwords off-site as well. Suppose my house burns to the ground, I need a way to retrieve the data - otherwise an off-site backup is pretty useless. And to make things life easier in case of emergency, I came up with the following idea:
  • A Debian 'live' USB stick (because, well, I'm a Debian fan)
  • Modified to contain my decryption keys for off-site backup
  • Modified to contain also scripts for retrieving the data
  • Modified to contain other data that could be useful, just in case.
  • Having a second partition so that data can be stored on the same USB stick
  • And since I plan to use this stick also for other emergencies: add a few packages like gparted and others.
  • Since I like things scripted, I did everything in a simple script for easy reproduction of the steps

What's in the script?
OK, going through the script line by line. Note that I'm not a Linux expert. Not a bash expert. Not a Debian expert. I'm 100% amateur. The result is very likely NOT fool-proof - but it did what I needed it to do.

Note that I execute all this as user 'root' - not sure if it could have been avoided,but the 'chroot' and 'mount' commands need root.

code:
1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

iso=$(wget -O - -o /dev/null https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid |
     grep amd64-gnome.iso |
     sed 's/^.*a href="//;s/".*$//')
if [[ (${?} -ne 0) ]]; then
  echo "couldn't determine Debian ISO filename"
  exit
fi

newiso=$(echo ${iso} | sed 's/gnome/gnome-personal/')

We start everything by determining what the latest regular Debian Live ISO is. I've selected AMD64 as architecture, which is what I need pretty much always. And also the gnome desktop environment, for no good reason. After above code, hopefully variable $iso contains the filename of the image. And variable $newiso already contains the target filename.

code:
1
2
3
4
5
6
7
if [ ! -f ${iso} ]; then
  wget "https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/${iso}"
  if [[ (${?} -ne 0) ]]; then
    echo "Couldn't download iso ${iso}"
    exit
  fi
fi

Next, the live ISO is downloaded. To make development of this script a bit faster, I first check if the iso was already downloaded before (and then do NOT download it again).

code:
1
2
3
4
5
dd if=${iso} bs=1 count=432 of=isohdpfx.bin
if [[ (${?} -ne 0) ]]; then
  echo "dd couldn't extract isohdpfx.bin from iso"
  exit
fi

If I understood correctly, the first 432 bytes of the downloaded ISO contain the master boot record (MBR). We need to extract it, so that we can use it again, later on in the script, when creating the target ISO.

code:
1
2
3
4
5
6
mkdir -p cd
xorriso -osirrox on -indev ${iso} -extract / cd
if [[ (${?} -ne 0) ]]; then
  echo "xorriso failed to extract from iso"
  exit
fi

This will extract the files in the ISO to a directory 'cd' - so this directory contains all the files of the Debian live image. This is not just for the Linux 'live' environment, but also for installing Debian to a new system. And we're now mostly interested in modifying the 'live' environment. So...

code:
1
2
3
4
5
6
7
rm -fr squashfs
mkdir squashfs
unsquashfs -f -d squashfs cd/live/filesystem.squashfs
if [[ (${?} -ne 0) ]]; then
  echo "unsquashfs failed to extract from filesystem.squashfs"
  exit
fi

...we're creating a squashfs directory and unpack the complete 'live' environment into this directory. Following this, we can start modifying it:
code:
1
2
cp -v --dereference /etc/resolv.conf squashfs/etc/resolv.conf
sed -i.bck 's/main *$/main contrib non-free/' squashfs/etc/apt/sources.list

The first line is needed so that also from within a chroot environment the DNS server can be found.
The second line is because I need a non-free repository as well - I'm including the Intel wifi drivers, just in case I'm using this ISO on my laptop and I need WiFi connectivity.

code:
1
2
3
4
5
6
7
8
9
mount -o bind /proc    squashfs/proc
mount -o bind /dev     squashfs/dev
mount -o bind /dev/pts squashfs/dev/pts
mount -o bind /sys     squashfs/sys
mount -o bind /run     squashfs/run
chroot squashfs /bin/bash -c "apt -y update && apt -y upgrade && apt -y autoremove"
chroot squashfs /bin/bash -c "apt -y install duplicity python3-b2sdk gparted firmware-iwlwifi"
chroot squashfs /bin/bash -c "apt-get clean"
umount squashfs/proc squashfs/dev/pts squashfs/dev squashfs/sys squashfs/run

Lots of mounts so that we can do chroot. Then, within a chroot environment, first update the packages to the latest available, then install whatever is needed. For retrieving my off-site backup data, I at least need duplicity and python3-b2sdk. I also want to be able to use this stick as partition editor (so added gparted) and should be able to use it on my laptop (so need firmware-iwlwifi, which is non-free). To limit the ISO size, the apt cache is cleaned. Following this, we can unmount the mounted directories.

This part and the next part are probably where you want to make your own choices.

code:
1
2
3
4
5
6
mkdir -p squashfs/home/user/bin
cp -r .duplicity_keys                          squashfs/home/user/
cp    bin/run_duplicity*                       squashfs/home/user/bin/
chown -R 1000:1000 squashfs/home/user

rm squashfs/etc/resolv.conf

After booting the live environment, you're working as user with username 'user' so a home directory /home/user will be used. We can already create this directory (and lower levels), and can already predefine the content. So: the keys for the off-site backup is already copied to the user directory and also the various scripts I created to backup/restore to/from the off-site location are copied to a bin directory. Then the copied data is chown'ed to UID 1000 - which matches the live user UID. Finally the resolv.conf file is removed again, as it is not needed in the live environment (assuming DHCP will provide DNS resolving).

Of course above can be expanded to include other data as well.

So, we're done defining the content of the target ISO. Now to create the image is basically creating the reverse steps from extracting the ISO:
code:
1
2
mv cd/live/filesystem.squashfs filesystem.squashfs.ori
mksquashfs squashfs cd/live/filesystem.squashfs -comp xz

The original 'live' environment is first moved away from the directory with ISO files. I could also just have removed it, but this works also for me. Then, the modified live environment is packed into a new squashfs file. Using the 'xz' compression results in a nice compact file - the original ISO also used this compression format.

code:
1
2
3
4
5
6
7
8
9
10
chmod -R -w cd
xorriso -as mkisofs \
        -o ${newiso} \
        -isohybrid-mbr isohdpfx.bin \
        -c isolinux/boot.cat \
        -b isolinux/isolinux.bin \
        -no-emul-boot \
        -boot-load-size 4 \
        -boot-info-table \
        ./cd

Now the tool xorriso is used to create the target ISO - using the updated live image, the filesistem.squashfs somewhere within the cd directory. And using the MBR as extracted earlier from the original ISO.

At this point, the directories cd, squashfs and files isohdpfx.bin and filesystem.squashfs.ori can be removed - or kept, if you want to investigate a bit more.

The ISO can then be written to a USB stick with e.g. the 'rufus' tool. Within this tool's GUI, you can slide the 'persistent' slider so that any remaining USB storage is converted into a second partition - which you can use e.g. when retrieving the off-site backup. That's then created with an ext4 filesystem - So not well readable from e.g. Windows. I've then, from within the windows disk manager, deleted the partition and created an NTFS partition instead.

When testing the ISO, the NTFS partition was automatically mounted (sorry, forgot the mount location - but a 'mount' command in a terminal shows where it is mounted) and I could retrieve data from the off-site backup to the NTFS partition. From then on, the data on this partition was accessible also on a Windows system. Mission accomplished! :)


Final words
Of course I didn't invent all this myself. The internet is full of similar guides for various uses, with different distributions. This is what I came up with for my personal needs - and put it in this blog in case it is useful for others. :) Note that the first part of the script does nice error checking and exiting on errors... but I got tired of doing that later on.

When trying all this... do NOT forget to (a) test the result, and (b) store the USB stick at a safe external location!

And disclaimers: use at own risk. I'm not responsible for any damage. And so on.

Comments


By Tweakers user eXcyle, vrijdag 15 oktober 2021 00:37

code:
1
echo "nameserver 192.168.1.1" > squashfs/etc/resolv.conf

Not sure why the first line is needed - but if I don't add this, running 'apt' in the chroot environment later on doesn't work because it cannot resolve the URL for the Debian repositories.
when you use the chroot command, you change the root directory ( / and all relative subdirectories beneath it) for the command that follows to the directory you specified ( /root/squashfs/ ).

code:
1
chroot squashfs /bin/bash -c "apt -y update && apt -y upgrade && apt -y autoremove"

so using this command you start /bin/bash and set/redirect the root directory to be your /root/squashfs directory

When the apt command runs within the bash shell you just spawned, it tries to locate your nameserver by reading the /etc/resolv.conf file, but because of the chroot, it is actualy reading the content from /root/squashfs/etc/resolv.conf, and not the /etc/resolv.conf file from your normal OS filesystem.

Without the chroot command apt would install the software into your actual system and not the new LiveCD filesystem.

You might also want to remove the nameserver line from the "/root/squashfs/etc/resolv.conf" file when you are done, otherwise the nameserver would be hardcoded into the new live cd (it should get overwritten by the dhcp-service, but still

[Comment edited on vrijdag 15 oktober 2021 00:42]


By Tweakers user vanaalten, vrijdag 15 oktober 2021 09:01

@eXcyle
Clear, thanks! And indeed, just tested the live image and the nameserver is hard coded in /etc/resolv.conf - so better to remove it.

[Comment edited on vrijdag 15 oktober 2021 09:01]


By Tweakers user sfranken, vrijdag 15 oktober 2021 11:11

Don't use backticks anymore, that's been deprecated for quite a while. Use $(...) for shell commands and ${...} for variables, like so:

code:
1
2
3
4
5
6
7
8
9
10
#!/bin/bash

iso=$(wget -O - -o /dev/null https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid | grep amd64-gnome.iso | sed 's/^.*a href="//;s/".*$//')

if [[ ${?} -ne 0 ]]; then
    echo "Couldn't determine Debian ISO filename."
    exit 2
fi

newiso=$(echo ${iso} | sed 's/gnome/gnome-personal/')


For the resolv.conf thing, you can also copy the current one:

code:
1
sudo cp -v --dereference /etc/resolv.conf squashfs/etc/resolv.conf

[Comment edited on vrijdag 15 oktober 2021 11:16]


By Tweakers user vanaalten, vrijdag 15 oktober 2021 11:28

@sfranken
"Don't use backticks anymore, that's been deprecated for quite a while."
I know. I just can't get used to the $(...) notation. Too much typing! ;)

But good point about copying the current resolv.conf - makes the script at least easier to reuse by others. I'll probably update the blog somewhere in the near future with all these minor changes.

Comments are closed