[How to do instruction] Making your Pi Read-Only

[jbaumann]

 
Why?
One of the common usage scenarios for the Moode player (and other players) is to connect its power to the same switched power strip that turns the stereo amplifier on and off. In this scenario the Raspberry Pi cannot orderly shut down, because its power is cut from one moment to the next.
Now your question might be: So what? The worst that can happen with modern file system is that a few bytes won‘t be written to the sd card and that‘s it.
But that is not really true. The problem here is that as a difference to common hard disks writing to an sd card (or even SSDs for that matter) doesn‘t mean you only write the 512 bytes of the current sector (or even the 4K) that hold parts of the current file. Depending on the controller circuitry and the internal organization of the sd card (or SSD) you first read up to 32K of non-contiguous sectors (the internal organization and the external representation are totally different), modify the one sector you want to writ to, and then write the same non-contiguous sectors back to the sd card (or SSD). This absolutely makes sense from a number of view points, but in case of power loss this leads to the problem that a power loss during a write might destroy your current installation.
You might say: So what? I have a backup and simply write the last version on the sd card again, and all is well.
And in most case you‘d be right. But there is a small „window of opportunity“ for the sd card (or SSD) controller (which manages all reads and writes) to foobar not only the installation, but in fact destroy the whole block of up to 32K of data.
And again you ask: So what? In that case I reformat the sd card and use it with a little bit less capacity.
And in most cases you are right. But in a small number of cases the whole sd card (or SSD) is fried. And believe me, this is not hearsay, I have killed multiple sd cards and one SSD myself (but then I‘m not the typical user).
But even now you could argue, that in such cases you simply write your backup to a new sdcard and be done.
And you are absolutely right.
Two arguments though:

• If you are an OCD nerd like me, then you don‘t like a Linux system being cut from power without proper shutdown
  
• It is always fun to learn new ways to abuse your Linux systems while keeping them happy at the same time.
  

TL;DR: This is not exactly necessary if you accept that from time to time you need to buy a new sd card. As long as you have a backup. You have a backup, right?
 
The Idea
For quite some time Raspbian has offered to turn your system into a read-only system that writes all changes to a temporary RAM disk. [Side note: This is much better than before, when you had to jump through a number of hoops to get to a stable read-only system]. By redirecting all changes to a RAM disk, no writes happen and your sd card (or SSD) is safe.
The advantages are clear: Your sd card cannot be harmed, if no writes happen. Every time when you boot, your system returns to the exact same state it was in when you turned on the read-only file system.
The disadvantages follow directly from this: Every change you make is temporary, after a reboot every change is lost, and in addition, since the RAM disk is stored in RAM (duh), only a limited number of changes can happen until the RAM disk is full. Pllease remember, writing to the system logs is a change, and Linux normally is quite verbose.
So after some time you have to reboot your system, simply because the RAM disk is full. With an unchanged Moode player installation, you can expect between 9-12 hours on a RPi3 until this happens (the RPi3 has 1GB of RAM). On an RPi4 with more RAM you get even longer times. So for all practical purposes there is no problem. But we can optimize the system regarding its disk writes, and thus get even longer times (see optimization for that). And we could even install a daemon that watches the RAM utilization and automatically reboots when the RAM disk is full.
But what about not being able to change anything? If we want to update the system, change the library or playlists, we simply switch back to the „normal“ installation (that stores all changes to the sd card), do our updates/changes, and then turn the read-only system back on.
 
How
The functionality we are talking about is called OverlayFS and can be turned on and off using the program raspi-config in standard Raspbian distributions. Using the program instead of scripting the functionality on our own means that with every update to that program we automatically use the newer and better implementation to turn the OverlayFS on and off.
 
Since the Moode Player is based on a standard Raspbian distribution, we can simply use the functionality Implemented in raspi-config. But instead of going through the graphical interface we will use the command-line interface that allows to call functions directly instead of scrolling through menus.
 
We will add a few simple scripts that use this command-line interface to simplify our life.
 
For this we first have to login via ssh, create a directory ~/bin and cd into it. Then we create each of the following commands with the editor of our choice, and finally change its mode to „executable“.
 
This has the advantage that starting  with the next login these scripts will be available as normal commands.
 
Let us review the commands:

Code:

> ssh <user@moode>

If you use putty on Windows, or iTerm on OSX, do the equivalent…

Code:

> mkdir bin
> cd bin

Now we are ready to add all the scripts. 

Code:

> vi enable_overlayfs

 Now paste the contents of the file as listed below into that file

Code:

> chmod +x enable_overlayfs

 
Repeat these last two steps for all the scripts.
 
The last step is to read every one of the scripts, understand what is happening and why. Do not trust me or this text.
 
Now you are ready to log out and log in again. After you have done that you can use these scripts like any other command. Even before you can use them by explicitly referencing them with their full path i.e., by prefixing ~/bin/<command>.
 
After you have logged in again, configure your system exactly like you want to have it start-up (including a radio station or playlist playing, equalizer settings etc.). The final step is to call „enable_overlayfs“, check the output of „check_overlay“ and reboot.
 
enable_overlayfs

Code:

#!/bin/sh

sudo raspi-config nonint enable_overlayfs
sudo raspi-config nonint enable_bootro

This script uses raspi-config in its non-interactive mode to first enable the OverlayFS and then switch the boot partition to read-only mode. To activate the OverlayFS we need to reboot after executing the script.

disable_overlayfs

Code:

#!/bin/sh

sudo raspi-config nonint disable_overlayfs

This script uses raspi-config to disable the OverlayFS. To actually turn it off we have to reboot the system.
 
remount_boot_rw

Code:

#!/bin/sh

sudo mount -o remount,rw /boot

This script simply remounts the boot partition as read/write so that we can change it. We do not change the entry in the fstab, because we want the boot partition to be writable only temporarily. After the next boot it should be back to read-only mode.
 
This is important when we want to change boot parameters, the kernel configuration or when we update the system with apt (since it might want to write e.g., a new kernel). Don't worry though, if you forget to remount the boot partition as read/write, apt will throw errors as soon as it tries to write to /boot, which in turn reminds you to remount it.
 
check_overlayfs

Code:

#!/bin/sh

if grep -q "boot=overlay" /boot/cmdline.txt; then
  echo -n "Overlay is enabled "
else
  echo -n "Overlay is disabled "
fi

if grep -q "boot=overlay" /proc/cmdline; then
  echo "and is on"
  df|grep "overlay\|Filesystem"
  echo
else
  echo "and is off"
fi

if findmnt /boot | grep -q " ro,"; then
  echo "Boot partition is mounted ro"
else
  echo "Boot partition is mounted rw"
fi

if grep -q "/boot.*vfat.*\(,ro\|ro,\)" /etc/fstab; then
  echo "/boot in /etc/fstab is ro"
else
  echo "/boot in /etc/fstab is rw"
fi

This is by far the most complicated script, and it simply prints out the current status of the OverlayFS and its utilization, the current mount status of the boot partition and whether it is set to read-only in fstab (the file system table).
 
Optimizations
There is one very simple optimization that can be implemented with one command. By replacing the normal logging functionality (the rsyslog daemon) with another one that logs only to a fixed-size RAM buffer, the number of writes to the normal file system can be significantly reduced.
The advantage is the fixed size that initially takes more RAM than simply writing the changes to the OverlayFS, but quickly reduces the overall RAM consumption. Changing this leads to a RPi3 being able to run for weeks without problems.
One important point: This only changes the logging for all those programs that use the normal syslog approach (Moode itself as well as some other programs don‘t do this and their logs will be found in the usual place).
The alternative log daemon is the busybox-syslogd, which is extremely stable and is in use in countless distributions. This means that switching to this implementation introduces no additional risks or instabilities.
The command to install the daemon is as follows:

Code:

> sudo apt install busybox-syslogd

 This installs the daemon and automatically switches the logging to RAM only. That‘s it.
 
One disadvantage though: You have to look at your logs using „logread“ or „logread -f“ (for continous output) instead of the usual way to browse your logs. But that is, in my opinion, a small price to pay.
 
 
Recipes
 
Updating the OS/Player
First check the state of OverlayFS and boot partition with „check_overlayfs“.
If the OverlayFS is turned on and enabled at boot, call „disable_overlayfs“. Reboot.
Call „check_overlayfs“.
If the OverlayFS is turned off and the boot partition is mounted read-only, call „remount_boot_rw“.
Now the system is in a state in which it can be updated.
> sudo apt update; sudo apt upgrade
 
These two commands update your whole installation with all packages, including the Moode Player.
 
After this is done, check that your player configuration is exactly like you want to have it after a boot. Then call „enable_overlayfs“. Reboot.
 
Changing the configuration
First check the state of OverlayFS and boot partition with „check_overlayfs“.
If the OverlayFS is turned on and enabled at boot, call „disable_overlayfs“. Reboot.
Call „check_overlayfs“.
The OverlayFS should be turned off
Now the system is in a state in which it can be changed.
 
Change the configuration, install new packages (if they need to change the boot partition call „remount_boot_rw“ first) and do what has to be done.
 
After you are done, check that your player configuration is exactly like you want to have it after a boot. Then call „enable_overlayfs“. Reboot.
 
Final Words
Yes, it is that simple to use your Moode Player in a read-only installation. Have fun with it.