The Triple Threat - Raspberry Pi 4b, Ubuntu 20.04.1 64-bit, and SSD Boot

Over the course of 2020, there was much enthusiasum in the Raspberry Pi community over the evolving ability of the Raspberry Pi 4b model's to boot its operating system 'directly' from an external storage device. The ride has been a bit bumpy along the way, but very manageable for the hobbyist to play along.

However, I wanted to bring together four specific components to the equation:

  • Ubuntu Server 20.04 LTS 64-bit
  • an external SSD as the primary/OS storage device
  • and of course, a Raspberry Pi 4b (with non-beta firmware)
  • And most importantly, I didn't want to have to do a lot of 'afterwork' to maintain the setup.

Specifically, I didn't want to have to manually copy a bunch of RPi-specific boot files around to support booting a 64-bit kernel, nor with decompressing the compressed kernel that is generated by default when the Ubuntu kernel is updated in the normal course of maintenance. I believe that I may have waited just long enough for the Raspberry Pi firmware to have sufficiently matured to solve the former concern, while the scripting shown below will solve the latter concern.

I'm going to be using Raspberry Pi OS Lite, Release Date December 2, 2020 to get things started/prepped, and Ubuntu Server 20.04.1 LTS 64-bit for Raspberry Pi 4 as the final objective OS that will boot/run from the SSD.

BTW, as a personal preference, I'll be performing these steps on my workstation that is currently running Ubuntu Desktop 20.04.1 LTS 64-bit. I'll also exclusively use shell commands throughout and ssh to the Raspberry Pi as soon as possible in the process.

Download and Checksum verification of OS Images

I downloaded the latest Raspberry Pi OS Lite and latest 64-bit Ubuntu Server 20.04.1 LTS for Rasperry Pi 4 images to my workstation.

Using this easy tutorial, I verified the checksum of the downloaded Raspberry Pi OS .zip file, then used unzip to extract the .img file from the .zip in preparation for writing it to the SD card.

me@workstation:~/Downloads$ unzip 2020-12-02-raspios-buster-armhf-lite.zip 
Archive:  2020-12-02-raspios-buster-armhf-lite.zip
  inflating: 2020-12-02-raspios-buster-armhf-lite.img

The Ubuntu image comes down as an ...img.xz file, which is just a compressed .img file. In order to decompress it on the command line, you may need to install xz-utils. I then used unxz to uncompress the img.gz file to an .img file. Now both .img files are ready for writing to their respective media.

me@workstation:~/Downloads$ unxz ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz

Note: I used sha256sum to calculate SHA-256 hashes of each of the files, and added the results to a 'checksum' file in my Downloads folder so that in the future, I can simply run 'sha256sum --check checksum' to verify that those images are unmodified/correct.

Now we have both images ready for writing out to boot media for the Raspberry Pi.

Imaging Raspberry Pi OS to the SD, First boot, and Updates

Once downloaded, verified, and decompressed (and perhaps verified again), the Raspberry Pi and Ubuntu OS images are ready for writing to their respective boot media.

Being careful with what we aim dd at, and when

I will use an 8Gb Micro SD card to boot the Rasbperry Pi OS. After inserting that SD card, my workstation automatically scans the new drive and mounts any existing partitions. A quick (edited) perusal of the output of df shows that my 8Gb SD card was mounted as /dev/sdc with two partitions.

me@workstation:~/Downloads$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc2       7.1G  1.4G  5.4G  21% /media/me/rootfs
/dev/sdc1       253M   54M  199M  22% /media/me/boot

I'll be overwriting the entire contents of /dev/sdc, but first I'll unmount these partitions so as not to throw my workstation into confusion.

me@workstation:~/Downloads$ sudo umount /dev/sdc1
[sudo] password for me: 
me@workstation:~/Downloads$ sudo umount /dev/sdc2

A quick re-check with df will show that these partitions are now unmounted from the OS and we can safely proceed to writing the image to the device.

Note: Be sure not to eject the SD. We still need the device to remain connected to the USB bus for what we are about to do next. You can check that the device is still attached by using the lsblk command.

me@workstation:~/Downloads$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
... { a bunch of snap loops omitted from this copy of the output }
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0 232.4G  0 part 
├─sda2   8:2    0   516M  0 part 
├─sda3   8:3    0   513M  0 part /boot/efi
├─sda4   8:4    0     1K  0 part 
└─sda5   8:5    0 232.4G  0 part /
sdb      8:16   0 931.5G  0 disk 
└─sdb1   8:17   0 931.5G  0 part /mnt/3ECA-07DD
sdc      8:32   1   7.4G  0 disk 
├─sdc1   8:33   1   256M  0 part 
└─sdc2   8:34   1   7.2G  0 part 
sr0     11:0    1  1024M  0 rom

The workstation still sees /dev/sdc and its partitions, but they are not mounted to the file system tree of the OS.

Writing Raspberry Pi OS to the SD card

Now we simply issue the dd command to write the image to the device file, and THEN eject the SD card from the workstation. Be sure to confirm that you have the correct drive identifier as the argument for of= or your day can go very badly.

me@workstation:~/Downloads$ sudo dd if=2020-12-02-raspios-buster-armhf-lite.img of=/dev/sdc bs=1M
1772+0 records in
1772+0 records out
1858076672 bytes (1.9 GB, 1.7 GiB) copied, 266.099 s, 7.0 MB/s
me@workstation:~/Downloads$ sudo eject /dev/sdc

A final lsblk check will show that /dev/sdc/ is no longer in the USB hardware graph, and we can safely unplug the SD card from the workstation.

Writing Ubuntu OS to the SSD

I will use a 500Gb USB3 SSD drive to eventually serve as the primary/only boot device for my Raspberry Pi 4. After plugging in the SSD drive, my workstation automatically scans the new drive and mounts any existing partitions. A quick (edited) perusal of the output of df shows that similar to above with the SD card, this USB SSD was mounted as /dev/sdc with two partitions.

me@workstation:~/Downloads$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc1       253M  143M  110M  57% /media/me/system-boot
/dev/sdc2       459G  2.7G  438G   1% /media/me/writable

These partitions must also be unmounted, just as was done for the SD card. We'll then write out the Ubunto OS image to the SSD using dd and eject it.

me@workstation:~/Downloads$ sudo umount /dev/sdc1
[sudo] password for me: 
me@workstation:~/Downloads$ sudo umount /dev/sdc2
me@workstation:~/Downloads$ sudo dd if=ubuntu-20.04.1-preinstalled-server-arm64+raspi.img of=/dev/sdc bs=1M
3100+1 records in
3100+1 records out
3250824192 bytes (3.3 GB, 3.0 GiB) copied, 95.9733 s, 33.9 MB/s
me@workstation:~/Downloads$ sudo eject /dev/sdc

We now have both of our OS images ready for first use on the Raspberry Pi 4b.

First (and second) Boot of the Rasbperry Pi OS

My Raspberry Pi 4b has a the 'fresh' SD card prepared just above, now inserted along with a keyboard, mouse, and HDMI display connected to the RPi. Do not plug in the SSD to the RPi just yet. Also, the RPi is not connected to any ethernet, but I will be attaching that shortly. The first thing that I want to do is successfully boot the RPi, and change the default password.

The RPi does its usual business of a first boot, followed by an immediate File System Resize for the main partition, and then the second, full boot. After trying, and (intentionally allowing it to) failing to connect to a network, we get the expected raspberrypi login: prompt and we enter the default login credentials of pi/raspberry.

After logging in, issue the passwd command to change the password for user pi. Now I'd like to enable ssh on the RPi by creating an empty file named ssh in the /boot directory, plug in my ethernet cable, and then reboot.

pi@raspberrypy:~ $ sudo touch /boot/ssh
pi@raspberrypy:~ $ sudo shutdown -r now

While the RPi reboots, I notice on my HDMI-connected terminal that the IP address assigned to my ethernet port is 192.168.0.223, and I log in using ssh from my workstation.

me@workstation:~$ ssh pi@192.168.0.223
The authenticity of host '192.168.0.223 (192.168.0.223)' can't be established.
ECDSA key fingerprint is SHA256:X9Bu+Tsu8XnLoB+qiTuj6imOiRFcDcOWoVUN5iyHFCE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.223' (ECDSA) to the list of known hosts.
pi@192.168.0.223's password: 
Linux raspberrypi 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Dec  2 12:58:32 2020

Wi-Fi is currently blocked by rfkill.
Use raspi-config to set the country before use.

pi@raspberrypi:~ $

Now we'll update the system and all of the installed applications.

pi@raspberrypi:~ $ sudo apt-get update && sudo apt-get dist-upgrade -y

Note: We could spend some time using raspi-config to set locale, wifi country, US keyboard, etc. but since we're not planning to use this SD card for much more than managing the RPis firmware.

For now, we will NOT reboot the RPi, but rather, move on to changing the revision stream for the RPi firmware releases.

Choosing the Firmware Release Stream for the Rasbperry Pi OS

The official Raspberry Pi documentation page for the Raspberry Pi 4 boot EEPROM (aka Firmware) provides additional details to the examples below.

Next, we'll edit the rpi-eeprom-update file to pick up the 'stable' stream of firmware releases. As of December, 2020, it's not necessary to ride the 'beta' path. In fact, we could remain on the 'critical' path, but for illustrative purposes, we'll select the 'stable' path to demonstrate some of the firmware management tools.

The single-line contents of the rpi-eeprom-update file determines the release stream for firmware updates. By default, it is set to 'critical' which is the most conservative release path.

pi@raspberrypi:~ $ cat /etc/default/rpi-eeprom-update
FIRMWARE_RELEASE_STATUS="critical"

Let's change that to 'stable' using the command line editor, 'vi'.

pi@raspberrypi:~ $ sudo vi /etc/default/rpi-eeprom-update

FIRMWARE_RELEASE_STATUS="stable"

We can check the currently installed/active firmware using the command below:

pi@raspberrypi:~ $ vcgencmd bootloader_version
Sep  3 2020 13:11:43
version c305221a6d7e532693cc7ff57fddfc8649def167 (release)
timestamp 1599135103
update-time 0
capabilities 0x00000000

Now that the stream has changed from 'critical' to 'stable', we can check if a newer version of firmware is available on the 'stable' path, and in fact, there is one available.

pi@raspberrypi:~ $ sudo rpi-eeprom-update
BCM2711 detected
VL805 firmware in bootloader EEPROM
*** UPDATE AVAILABLE ***
BOOTLOADER: update available
CURRENT: Thu  3 Sep 12:11:43 UTC 2020 (1599135103)
 LATEST: Fri 11 Dec 11:15:17 UTC 2020 (1607685317)
 FW DIR: /lib/firmware/raspberrypi/bootloader/stable
VL805: up-to-date
CURRENT: 000138a1
 LATEST: 000138a1

Let's reboot and confirm that the new 'December 11, stable' firmware is installed in place of the current 'September 3, critical' firmware.

Note: I believe that the steps above SHOULD have caused the firmware to update simply by rebooting having changed the value to 'stable' in the config file, however, it did not. I needed to execute the following commands (none of which indicated via output text to screens that any change would be made), in order to get the EEPROM updated to the December 11 revision. shrug.

sudo apt update
sudo apt full-upgrade
sudo reboot

After logging in, we can see that the Dec 11 version is indeed installed after this second reboot.

pi@raspberrypi:~ $ vcgencmd bootloader_version
Dec 11 2020 11:15:17
version c3f26b6070054bca030366de2550d79ddae1207a (release)
timestamp 1607685317
update-time 1609352305
capabilities 0x0000001f

Ok, enough of that. Recall that the 'critical' firmware path is sufficient for the purposes of this example, and worked perfectly for me in drafting this tutorial.

On to Ubuntu...

Introducing the Ubuntu Boot SSD to the Raspberry Pi

Earlier in this tutorial, we wrote the latest image of Ubuntu 20.04 Server LTS 64-bit to the SSD. We're now ready to prepare it for use to boot our Raspberry Pi 4b.

Plugging the SSD in to one of the USB3 ports on the Raspberry Pi, then checking the hardware graph for the connected block storage devices, we'll find our new SSD.

pi@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 465.8G  0 disk 
├─sda1        8:1    0   256M  0 part 
└─sda2        8:2    0   2.8G  0 part 
mmcblk0     179:0    0   7.4G  0 disk 
├─mmcblk0p1 179:1    0   256M  0 part /boot
└─mmcblk0p2 179:2    0   7.2G  0 part /

We'll need to copy the .dat and .elf files from the Raspberry Pi /boot partition in to the Ubuntu's boot partition so that the Pi can successfully boot Ubuntu from USB.

First, we create mount points on the SD, mount the SSD's partitions to these mount points, and check that they are successfully mounted.

pi@raspberrypi:~ $ sudo mkdir /mnt/boot && sudo mount /dev/sda1 /mnt/boot
pi@raspberrypi:~ $ sudo mkdir /mnt/principal && sudo mount /dev/sda2 /mnt/principal
pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.1G  1.4G  5.4G  21% /
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  8.5M  3.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   54M  199M  22% /boot
tmpfs           788M     0  788M   0% /run/user/1000
/dev/sda1       253M   60M  193M  24% /mnt/boot
/dev/sda2       2.7G  1.8G  776M  70% /mnt/principal

Making Ubuntu bootable on the SSD

Now we’ll copy some of the contents (specifically, the .dat & .elf files) of the /boot partition of the Raspberry Pi OS SD to /mnt/boot which is the boot partition of the Ubuntu OS on the SSD.

pi@raspberrypi:~ $ sudo cp /boot/*.elf /mnt/boot/
pi@raspberrypi:~ $ sudo cp /boot/*.dat /mnt/boot/

The Rasbperry Pi boot loader is unable to load a compressed 64-bit kernel, so we need to decompress it, however, we'll encounter a permissions issue in writing to the SSD's '/boot' partition unless we create a password for the 'root' user.

Therefore, the following steps are neccesary when starting from scratch and no previous root password has been set. Otherwise when running "zcat vmlinuz >vmlinux", a permission denied error occurs.

# Create root new password
pi@raspberrypi:~ $ sudo passwd
New password: 
Retype new password: 
passwd: password updated successfully
# Become root
pi@raspberrypi:~ $ su
Password: 
# decompress the Ubuntu kernel
root@raspberrypi:/home/pi# zcat /mnt/boot/vmlinuz > /mnt/boot/vmlinux
# quit being root
root@raspberrypi:/home/pi# exit
exit
pi@raspberrypi:~ $

Now we need to edit the [pi4] section in config.txt (found in /mnt/boot) as follows: sudo vi /mnt/boot/config.txt replacing the entire [pi4] section with the following:

[pi4]
max_framebuffers=2
dtoverlay=vc4-fkms-v3d
boot_delay
kernel=vmlinux
initramfs initrd.img followkernel

Keeping Ubuntu Bootable on the Raspberry Pi

The Ubuntu kernel is routinely updated/recompiled as part of its normal maintenance. Unfortunately, for Rasbperry Pi, Ubuntu kernels are compressed by default, therefore after each update to the kernel by apt, the Rasbperry Pi would no longer be able to boot. In order to solve this situation, we need to create a script to check if the Ubuntu kernel has been updated, and if so, to decompress it after the update.

We will create a script called auto_decompress_kernel in the boot partition. I used sudo vi /mnt/boot/auto_decompress_kernel to create this script containing the following code:

#!/bin/bash -e

# auto_decompress_kernel script

#Set Variables 
BTPATH=/boot/firmware 
CKPATH=$BTPATH/vmlinuz 
DKPATH=$BTPATH/vmlinux  

#Check if decompression needs to be done. 
echo -e "\e[32mChecking if kernel has been updated...\e[0m"
if [ -e $BTPATH/check.md5 ]; then  
   if md5sum --status --ignore-missing -c $BTPATH/check.md5; then
      echo -e "\e[32mFiles have not changed, Decompression not needed\e[0m"  
      exit 0  
   else 
      echo -e "\e[31mHash failed, kernel will be decompressed\e[0m"  
   fi 
fi

#Backup the old decompressed kernel 
mv $DKPATH $DKPATH.bak  
if [ ! $? == 0 ]; then  
   echo -e "\e[31mDECOMPRESSED KERNEL BACKUP FAILED!\e[0m"  
   exit 1 
else  
   echo -e "\e[32mDecompressed kernel backup was successful\e[0m" 
   echo "Decompressing kernel: "$CKPATH"..."
fi  


#Decompress the new kernel 
zcat $CKPATH > $DKPATH
if [ ! $? == 0 ]; then  
   echo -e "\e[31mKERNEL FAILED TO DECOMPRESS!\e[0m"  
   exit 1 
else  
   echo -e "\e[32mKernel Decompressed Succesfully\e[0m" 
fi  

#Hash the new kernel for checking 
md5sum $CKPATH $DKPATH > $BTPATH/check.md5

if [ ! $? == 0 ]; then  
   echo -e "\e[31mMD5 GENERATION FAILED!\e[0m"  
else 
   echo -e "\e[32mMD5 generated Succesfully\e[0m" 
fi

#Exit 
exit 0

and make it executable:

pi@raspberrypi:~ $ sudo chmod +x /mnt/boot/auto_decompress_kernel

Now we need to add a script to the apt update sequence that will call this script automatically as part of the apt update procedure. Add a script named 999_decompress_rpi_kernel in /mnt/principal/etc/apt/apt.conf.d/ with this command sudo vi /mnt/principal/etc/apt/apt.conf.d/999_decompress_rpi_kernel containing only the following line of code:

DPkg::Post-Invoke {"/bin/bash /boot/firmware/auto_decompress_kernel"; };

and make it executable:

sudo chmod +x /mnt/principal/etc/apt/apt.conf.d/999_decompress_rpi_kernel

Smile, you're (nearly) done.

At this point, the Rasbberry Pi 4b's firmware and bootloader configuration is set to successfully boot a 64-bit Ubuntu OS from the USB3-attached SSD drive. We also created a convenient, but essential, script to automatically update an uncompressed copy of the Ubuntu kernel each time it us updated by apt.

So now, power off the Raspberry Pi, remove the SD card, power it up and boot for the first time from your SSD drive.

As with Raspberry Pi OS, I'll be connecting to the Ubuntu OS on Raspberry Pi via SSH. Be sure to wait several seconds after the ubuntu login: prompt appears on the RPi's terminal. You'll know that you're good to go once you see the -----END SSH HOST KEY KEYS---- output on the terminal screen because this will mean that SSH keys are now available to respond to/negotiate an ssh connection.

Note: Recall that the default uid/passwd for ubuntu is "ubuntu/ubuntu".

Ubuntu will force you to change the password for user ubuntu and then immediately log you out. 

me@workstation:~$ ssh ubuntu@192.168.0.223
ubuntu@192.168.0.223's password: 
You are required to change your password immediately (administrator enforced)
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1015-raspi aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Dec 30 19:32:00 UTC 2020

  System load:           2.09
  Usage of /:            0.4% of 458.38GB
  Memory usage:          4%
  Swap usage:            0%
  Temperature:           75.0 C
  Processes:             146
  Users logged in:       0
  IPv4 address for eth0: 192.168.0.223
  IPv6 address for eth0: fdcb::fec1:f2c8
  IPv6 address for eth0: 2603::f2c8

 * Introducing self-healing high availability clusters in MicroK8s.
   Simple, hardened, Kubernetes for production, from RaspberryPi to DC.

     https://microk8s.io/high-availability

143 updates can be installed immediately.
75 of these updates are security updates.
To see these additional updates run: apt list --upgradable


Last login: Wed Dec 30 19:31:33 2020 from 192.168.0.6
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for ubuntu.
Current password: 
New password: 
Retype new password: 
passwd: password updated successfully
Connection to 192.168.0.223 closed.

So now log back in.

The first thing that I'd like to do is to veryify that the kernel decompression script is working properly as well as disable 'unattended-upgrades' which is active by default in the recent Ubuntu distributions.

ubuntu@ubuntu:~$ sudo apt remove unattended-upgrades
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  unattended-upgrades
0 upgraded, 0 newly installed, 1 to remove and 135 not upgraded.
After this operation, 451 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 66780 files and directories currently installed.)
Removing unattended-upgrades (2.3) ...
Processing triggers for man-db (2.9.1-1) ...
Checking if kernel has been updated...
Decompressed kernel backup was successful
Decompressing kernel: /boot/firmware/vmlinuz..............
Kernel Decompressed Succesfully
MD5 generated Succesfully

Now that unattended-upgrades won't be changing things on us unexpectedly in the future (although I may in-fact want to re-install it some other day), let's manually request these ~150 upgdates to be installed right now.

ubuntu@ubuntu:~$ sudo apt-get update && sudo apt-get dist-upgrade -y
...
135 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 222 MB of archives.
...
{ and eventually we reach...}
Taking backup of w5500.dtbo.
Installing new w5500.dtbo.
Checking if kernel has been updated...
Hash failed, kernel will be decompressed
Decompressed kernel backup was successful
Decompressing kernel: /boot/firmware/vmlinuz...
Kernel Decompressed Succesfully
MD5 generated Succesfully
ubuntu@ubuntu:~$

The next time that I logged in following a reboot, I did indeed notice that the kernel had incremented from '5.4.0-1015-raspi' to '5.4.0-1025-raspi'.

You now have a booting and maintainable 64-bit Ubuntu 20.04.1 LTS OS on a USB3-attached SSD on your Raspberry Pi 4b.

Now whatcha gonna do? :)

References

The following articles/tutorials were used to develop this guide, above. Raspberry Pi 4 Ubuntu USB Boot (No SD Card) No date given. Make Ubuntu server 20.04 boot from an SSD on Raspberry Pi 4 August 14, 2020.

social