Moode and power loss (aka writes to the sd card)

[jbaumann]

Hi Tim,

By turning of the rsyslog-service all logs can be turned off, since all logging from that point on is simply ignored.

Cheers, Joe

[Tim Curtis]

I think thats just for "system" logs.

Applications like moOde, MPD CamillaDSP, PHP, NGINX etc write to their own log files.

[jbaumann]

As long as they use the standard logging facility it is turned off as well.

But even for those that don‘t, they produce only a fraction of the logging the normal system does.

This means that by switching of the generic logging we get from e.g., 9-12 hours with OverlayFS on a RPi3 to something that I wasn‘t able to measure yet (days or weeks?)…

Cheers, Joe

[Tim Curtis]

Interesting.

I ran a quick test

Code:

sudo systemctl disable rsyslog.service
sudo /var/www/util/sysutil.sh clear-syslogs
sudo reboot

And end up with total 27M for /var/log/ which doesn't seem to grow

Code:

Every 2.0s: sudo du -h /var/log/                                                                                                                        moode: Sat Jun 24 10:21:30 2023

4.0K    /var/log/mpd
4.0K    /var/log/private
4.0K    /var/log/apt
4.2M    /var/log/sysstat
4.0K    /var/log/runit/dnsmasq
4.0K    /var/log/runit/ssh
12K     /var/log/runit
4.0K    /var/log/samba/cores/winbindd
8.0K    /var/log/samba/cores
16K     /var/log/samba
23M     /var/log/journal/8b6c504fa6fd4ec39aba2804b2479ff0
23M     /var/log/journal
4.0K    /var/log/nginx
27M     /var/log/

The systemd journal log (23M) from what I've read can't or shouldn't be turned off.

Couple notes:

1. Maybe add to the TODO list to have an option for System Logging (Default or Minimal) 
2. The built in moOde maintenance task runs every 6 hours and one thing it does is truncate the system logs using the command below.

Code:

/var/www/util/sysutil.sh clear-syslogs

[jbaumann]

My original idea was to reduce logging by changing the log level for the system logs. To see whether this approach would be able to yield anything at all I simply turned the logging off, and the result was astonishing for me. What would be the disadvantage to fully turn off the logging?

The problem with writing to the disk and truncating it is that the OverlayFS tries to follow all this and gobbles up RAM unnecessarily.

Would using busybox-syslogd be an alternative? This is a lightweight syslog daemon that writes to RAM and rotates/truncates automatically:

https://packages.debian.org/en/bullseye/busybox-syslogd

Cheers, Joe

[jbaumann]

This might just be the most elegant solution. It can be installed with

sudo apt install busybox-syslogd

and the configuration is done in /etc/defaults/busybox-syslogd (by default 128K of mem for the logs).

Cheers, Joe

[Tim Curtis]

My original idea was to reduce logging by changing the log level for the system logs. To see whether this approach would be able to yield anything at all I simply turned the logging off, and the result was astonishing for me. What would be the disadvantage to fully turn off the logging?

The problem with writing to the disk and truncating it is that the OverlayFS tries to follow all this and gobbles up RAM unnecessarily.

Would using busybox-syslogd be an alternative? This is a lightweight syslog daemon that writes to RAM and rotates/truncates automatically:

https://packages.debian.org/en/bullseye/busybox-syslogd

Cheers, Joe

Logging has never been a problem in the past and the built in maintenance tasks keeps log file size in check and so there is no practical reason for trying to disable it or add a new package that has to be maintained, plus we use logs all the time in troubleshooting and development. 

The old saying is that "logs are never needed until they are"

In your particular usage scenario though where you are using overlayFS and trying to create a memory only runtime for moOde it makes sense to try and reduce logging.

[jbaumann]

I now have written a guide for turning the Moode Player into a read-only installation.

Please find it here: https://moodeaudio.org/forum/showthread.php?tid=5678

I'm looking forward to your comments.

Cheers, Joe