Read-only root filesystem on Raspberry PI on Debian Buster

Once I asked a friend if he made his Raspberry PI based media player’s root filesystem read-only. He said that no, I can live with reinstalling it from time to time. Now my experience is that such failures happen the worst time possible: when you just want to watch a movie without wanting to go deep into a debug session or when you feel like your pet project is ready for a prime-time show for your friends.

The underlying issue is that whenever you unplug your Raspberry PI, it just interrupts any write to your SD card that is in progress. Of course, it is running Linux; as any proper operating system, that has a filesystem. Even though ext4 (the filesystem used in Raspbian by default) is designed in a way that it is able to tolerate power outages in-progress writes could cause file changes to be lost or inconsistent file contents.

Having a read-only root filesystem for your Raspberry Pi addresses this issue, but it also causes your root filesystem to be read-only. This is what you wanted right? Yes, but in reality, you probably want to write to your sd card from time to time: if you follow this guide you will also need to make sure that you can write to your SD card when you really want to. At the end of the post, you will find some tips on how to do this.

Setting it up

For this guide, I am assuming that you are using the Raspbian Duster Lite image available here. It is way easier to start from the Lite image than the desktop one, as using the steps detailed in the Verifying section you can always gradually add more software and make it read-only root compatible. Also if you want to use the Raspberry Pi as desktop, you might not want to have a read-only root filesystem, instead you should do clean shutdowns of your Pi.

I am also assuming that you are following this guide as being logged in as root user. To obtain a root shell you can execute the following command:

pi@raspberry:~$ sudo -i

Ideally, you should only have a handful of services that require write access to your SD card. The first one you will need to get rid of is rsyslogd. Let’s replace that with an in-memory logger:

$ apt-get update
$ apt-get install busybox-syslogd

# Execute this command if you want to check system logs 
$ sudo logread   

Unfortunately, systemd is not prepared to run on a read-only root filesystem, so let’s make some tweaks that will allow it to operate using read-only root filesystem:

# become root
$ sudo -i

# systemd-hostnamed uses a private temp folder on the root fs by default
# let's rather use /tmp
$ sed -i 's/PrivateTmp=yes/PrivateTmp=no/' /lib/systemd/system/systemd-hostnamed.service

# /tmp should be an in-memory filesystem
$ echo 'none    /tmp            tmpfs   size=128M,mode=01777 0 0' >> /etc/fstab
$ mount -a

# systemd tries to write last login/startup information to /var/log
# let's make it a tmpfs
$ echo 'none    /var/log         tmpfs   size=16M 0 0' >> /etc/fstab
$ mount -a 

Also, there are some system services that needs to be disabled:

# swapfile support must be disabled as it would rely on your root filesystem
$ systemctl disable dphys-swapfile.service

# disable daily apt-get update runs (make sure you run apt-get update manually)
$ systemctl disable apt-daily.service apt-daily.timer apt-daily-upgrade.timer apt-daily-upgrade.service

# disable logrotation as we don't have logs to be rotated
$ systemctl disable logrotate.service logrotate.timer

Verifying

Now let’s check if there are no open files: you will need to install a utility called lsof. This tool can be used to list the files opened in your system, so you can check which files are opened for writing.

$ apt-get update
$ apt-get install lsof

Then to check which files are open for writing you can execute the following command:

$ lsof / | awk 'NR==1 || $4~/[0-9]+[uw]/'
rsyslogd  321             root    7w   REG  179,2   358497  21334 /var/log/syslog
rsyslogd  321             root    8w   REG  179,2   170169  21336  

If the command outputs nothing, you are good to go: there are no files opened for writing on your too filesystem. In case the list is not empty (as the example shows above) the first column is the name of the application you need to deal with. You should consult the manual of that application on why it needs to have write access and either disable that functionality or put that onto a tmpfs volume.

Making the filesystems read-only

Now you should edit the /etc/fstab file, and append ro after defaults for / and /boot.

proc                  /proc           proc    defaults             0       0
PARTUUID=17869b7d-01  /boot           vfat    defaults,ro          0       2
PARTUUID=17869b7d-02  /               ext4    defaults,ro,noatime  0       1

Then reboot your Raspberry Pi using the shutdown -r now command.

After reboot login as root and check the output of this command (as root):

$ systemctl list-units --state=failed
0 loaded units listed. Pass --all to see loaded but inactive units, too.
 To show all installed unit files use 'systemctl list-unit-files'.

The above message is normal, it means that all of your system services had been successfully started (none of them is in failed state).

If you see an output like this:

$ systemctl list-units --state=failed
  UNIT                        LOAD   ACTIVE SUB    DESCRIPTION
● dphys-swapfile.service      loaded failed failed dphys-swapfile - set up, mount/unmount, and delete a swap file
LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type. 

Then please check why those units failed using the journalctl -u <UNIT-name-from-the-list> command.

Enabling writes

To enable writes for your root filesystem for maintenance tasks you can execute the following commands:

$ sudo mount / -o remount,rw
$ sudo mount /boot -o remount,rw

To go back to the read-only mode you just need to execute the following commands:

$ sudo mount / -o remount,ro -f
$ sudo mount /boot -o remount,ro -f

Feel free to create scripts in /usr/local/bin as convenience wrappers, or you can add these commands to your ~/.bashrc to automatically enable writes, then the read-only counterpart to your ~/.bash_logout file to remount all the filesystems in read-only mode when you are done administering your PI.

What if I cannot have a read-only root?

In this case, if you can live with the risk of corruption you should do nothing. In case you do want to mitigate the risk you can still change your /etc/fstab to something like this:

proc                  /proc           proc    defaults             0       0
PARTUUID=17869b7d-01  /boot           vfat    defaults,ro          0       2
PARTUUID=17869b7d-02  /               ext4    defaults,data=writeback,noatime  0       1 

This will make /boot read-only: that’s a VFAT filesystem that is not great at tolerating power outages. This also means that if you want to upgrade your kernel you will need to remount that in read-write mode (sudo mount /boot -o remount,rw).

For the root filesystem setting the data=writeback flag causes ext4 to be more strict on writing file contents (see the fstab manual for details). Warning: this setting also means that it will write all of the data twice to your SD card, meaning that it will wear faster.