[SOLVED] RPi3A+ Network instability.

[Tim Curtis]

Running a full update with htop running to see what happens.  Attached a screenshot taken from the frozen system.  Top window is the mpd log, the track being scanned is irrelevant, it doesn't fail on the same track each time.  The top process is nmap, that's interesting given the network oddness that goes with the freeze (network operations never timing out or failing , but also never completing either).

Nmap is used in a couple of places to scan the network on-demand for example
- Discover receivers 
- Players >>
- Scan for NFS shares

Its also used in the Mount Monitor to scan for NFS shares.

Try turning off Mount Monitor and also for good measure set Monitor to No for any Radio Stations that are being monitored. The station monitor follows the end of the mpd log for "alsa_output: Decoder is too slow; playing silence to avoid xrun" messages and then restarts MPD if there are more than a [threshold] number of them.

Another thing to do would be to set MPD logging to Verbose and also turn on moode debug logging. This would be to see if any of the extra detail in the logs has some clues to what might be happening.

[the_bertrum]

Turning off the mount monitor had no effect.  Verbose mpd logging showed that the apparently frozen state isn't actually frozen though, there are, very slowly, log entries, looping through this sort of thing:

Code:

un 24 19:46 : client: [225] process command "close"
Jun 24 19:46 : client: [225] command returned 4
Jun 24 19:46 : client: [225] closed
Jun 24 19:47 : client: [226] opened from 127.0.0.1:36500
Jun 24 19:47 : client: [226] process command "status"
Jun 24 19:47 : client: [226] command returned 0
Jun 24 19:47 : client: [226] process command "stats"
Jun 24 19:47 : client: [226] command returned 0
Jun 24 19:47 : client: [226] process command "close"
Jun 24 19:47 : client: [226] command returned 4
Jun 24 19:47 : client: [226] closed
Jun 24 19:47 : client: [227] opened from 127.0.0.1:55048
Jun 24 19:47 : client: [227] process command "status"
Jun 24 19:47 : client: [227] command returned 0
Jun 24 19:47 : client: [227] process command "stats"
Jun 24 19:47 : client: [227] command returned 0
Jun 24 19:47 : client: [227] process command "close"
Jun 24 19:47 : client: [227] command returned 4
Jun 24 19:47 : client: [227] closed

Any clue what that means?

[Tim Curtis]

Those queries to MPD are coming from the worker.php daemon (every 3 seconds) for example if CoverView screen saver is set something other than Never, Metadata file turned on, Library Update in progress, etc.

The function calls are if you want to Browse the code.
getMpdStatus()
getMpdStats()

This is all happening inter-process within the Pi and so not necessarily affected by a network issue. If these log messages appear regularly every 3 secs or so then it would suggest the processor is not pegged.

[the_bertrum]

Talk about barking up a red herring!

It wasn't network, it wasn't cpu, it was memory.  I searched about on the Raspberry pi forums and found a post that seemed familiar.  These folk were having bother with "apt-get upgrade y" on zero2 and 3a+ devices that didn't happen on other boards and they suspected it was 4 cores and 512 Mb memory that was the issue.  Increasing the swap file was their workaround, so I gave that a go. I made a 1Gb swap file and switched it on (moode defaults to off it seems).  Bingo every single scan runs to completion.  I used SMB and NFS with hostname, hostname.local and IP address variations.

Watching htop as the scan ran suggested that the swap file was unused until some winbindd and nfs scan processes kicked off.  That would explain the "networky" feel of the freeze maybe. htop also shows that the swap file maxxed out at 89 Mb, so probably the default 100 would be enough also.

Now what is it about my setup that consumes more memory than our esteemed Tim and Kent?  I wonder if it might be something in my library.  I have a mix of files from different places, FLAC, MP3 and M4A and I know that some that I inherited from my father have enormous images embedded.  I wonder if that could be it?

Anyhow, until either I sort my collection out and find the issue or the RPF release an update that helps the zero2 and 3A+ with memory use...  How can I make moode keep the swap file over a reboot, it keeps getting turned off.

[Tim Curtis]

Glad to hear a solution has been found. Sounds like a relatively easy fix too :-)

I'm not quite sure how swap works in Bookworm but I'll investigate tonight.
For a permanent solution it might be best to just automatically enable it during moode startup for < 1GB RAM devices.

[Tim Curtis]

To enable the swap file and make it permanent run the command below and then reboot.

Code:

sudo systemctl enable dphys-swapfile

sudo reboot

Verify its running after rebooting

Code:

systemctl status dphys-swapfile

free -m

Check the default configured size. It looks like it defaults to 200MB on <= 4GB RAM Pi's and 100MB on the 8MB RAM Pi's.

Code:

cat /etc/dphys-swapfile

It's currently being set to "disabled" in the post install section of the moode-player package. I'll just comment out that line for the upcoming 9.0.4 release and we should be all set.

[the_bertrum]

To enable the swap file and make it permanent run the command below and then reboot.

Code:

sudo systemctl enable dphys-swapfile

sudo reboot

Verify its running after rebooting

Code:

systemctl status dphys-swapfile

free -m

Check the default configured size. It looks like it defaults to 200MB on <= 4GB RAM Pi's and 100MB on the 8MB RAM Pi's.

Code:

cat /etc/dphys-swapfile

It's currently being set to "disabled" in the post install section of the moode-player package. I'll just comment out that line for the upcoming 9.0.4 release and we should be all set.

I got this locale error when enabling:

Code:

master@prometheus:~ $ sudo systemctl enable dphys-swapfile
Synchronizing state of dphys-swapfile.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable dphys-swapfile
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_TIME = "en_GB.UTF-8",
    LC_MONETARY = "en_GB.UTF-8",
    LC_ADDRESS = "en_GB.UTF-8",
    LC_TELEPHONE = "en_GB.UTF-8",
    LC_NAME = "en_GB.UTF-8",
    LC_MEASUREMENT = "en_GB.UTF-8",
    LC_IDENTIFICATION = "en_GB.UTF-8",
    LC_NUMERIC = "en_GB.UTF-8",
    LC_PAPER = "en_GB.UTF-8",
    LANG = "en_US.UTF-8"
   are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_TIME = "en_GB.UTF-8",
    LC_MONETARY = "en_GB.UTF-8",
    LC_ADDRESS = "en_GB.UTF-8",
    LC_TELEPHONE = "en_GB.UTF-8",
    LC_NAME = "en_GB.UTF-8",
    LC_MEASUREMENT = "en_GB.UTF-8",
    LC_IDENTIFICATION = "en_GB.UTF-8",
    LC_NUMERIC = "en_GB.UTF-8",
    LC_PAPER = "en_GB.UTF-8",
    LANG = "en_US.UTF-8"
   are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Created symlink /etc/systemd/system/multi-user.target.wants/dphys-swapfile.service → /lib/systemd/system/dphys-swapfile.service.

I also needed to start the service after the reboot:

Code:

sudo systemctl start dphys-swapfile

The default size on my Pi3A+ is 100MB, which is probably enough.  I'll test.

[TheOldPresbyope]

Swapping to uSD card is certainly faster than swapping to a hard drive so maybe a drag on moOde performance won't be noticed when swapping is enabled.

However, as an FYI, the default swappiness value in Raspberry Pi OS Lite---hence in a moOde installation--- is 60 which is rather aggressive given LInux's greedy swapping algorithm (e.g., more memory pages get swapped out during normal operation than is necessary just to avoid an OOM fault).

You can query the current swappiness value with the following command

Code:

sysctl vm.swappiness

I'd suggest cutting the value back to 10 (considered a typical workstation setting) or even 1 (a typical server setting).

This can be done by editing (as superuser) /etc/sysctl.conf to add a new last line vm.swappiness=10, say. and rebooting.

AsI said, perhaps the high value won't matter for moOde but since even on the low-memory Pis we seem to have issues only when indexing large libraries swapping is rarely needed. In the old days our Unix workstations would routinely slow to a crawl when swapping CAD processes to SCSI drives.

Just my 2-cents worth.

Regards,
Kent

[the_bertrum]

Swapping to uSD card is certainly faster than swapping to a hard drive so maybe a drag on moOde performance won't be noticed when swapping is enabled.

However, as an FYI, the default swappiness value in Raspberry Pi OS Lite---hence in a moOde installation--- is 60 which is rather aggressive given LInux's greedy swapping algorithm (e.g., more memory pages get swapped out during normal operation than is necessary just to avoid an OOM fault).

You can query the current swappiness value with the following command

Code:

sysctl vm.swappiness

I'd suggest cutting the value back to 10 (considered a typical workstation setting) or even 1 (a typical server setting).

This can be done by editing (as superuser) /etc/sysctl.conf to add a new last line vm.swappiness=10, say. and rebooting.

AsI said, perhaps the high value won't matter for moOde but since even on the low-memory Pis we seem to have issues only when indexing large libraries swapping is rarely needed. In the old days our Unix workstations would routinely slow to a crawl when swapping CAD processes to SCSI drives.

Just my 2-cents worth.

Regards,
Kent

It does feel as if there is something else in play here, and optimisations elsewhere might make enabling the swap file redundant.  What those optimisations might be however could be tricky.  The thread I linked to on the Raspberry Pi Forum seems to indicate that even a standard Pi OS doing an upgrade can run out of memory and need more swap, but conversely we have only one data point to show a moode installation that needs it.

More data on that front, a vanilla moode 9 with only NFS configured and everything else left default will scan my collection to completion with the default 100M swap.  My "full fat" configuration (Rotary Encoder, GPIO Buttons, Radio station monitor, CamillaDSP Loudness Volume, Spotify, UPnP, Bluetooth) managed to fill even that.  I tried again with 256M and all was well, free tells me the swap got to 107M.

Dare I suggest the swap file should be (yet another) option?

[Tim Curtis]

Swapping to uSD card is certainly faster than swapping to a hard drive so maybe a drag on moOde performance won't be noticed when swapping is enabled.

However, as an FYI, the default swappiness value in Raspberry Pi OS Lite---hence in a moOde installation--- is 60 which is rather aggressive given LInux's greedy swapping algorithm (e.g., more memory pages get swapped out during normal operation than is necessary just to avoid an OOM fault).

You can query the current swappiness value with the following command

Code:

sysctl vm.swappiness

I'd suggest cutting the value back to 10 (considered a typical workstation setting) or even 1 (a typical server setting).

This can be done by editing (as superuser) /etc/sysctl.conf to add a new last line vm.swappiness=10, say. and rebooting.

AsI said, perhaps the high value won't matter for moOde but since even on the low-memory Pis we seem to have issues only when indexing large libraries swapping is rarely needed. In the old days our Unix workstations would routinely slow to a crawl when swapping CAD processes to SCSI drives.

Just my 2-cents worth.

Regards,
Kent

It does feel as if there is something else in play here, and optimisations elsewhere might make enabling the swap file redundant.  What those optimisations might be however could be tricky.  The thread I linked to on the Raspberry Pi Forum seems to indicate that even a standard Pi OS doing an upgrade can run out of memory and need more swap, but conversely we have only one data point to show a moode installation that needs it.

More data on that front, a vanilla moode 9 with only NFS configured and everything else left default will scan my collection to completion with the default 100M swap.  My "full fat" configuration (Rotary Encoder, GPIO Buttons, Radio station monitor, CamillaDSP Loudness Volume, Spotify, UPnP, Bluetooth) managed to fill even that.  I tried again with 256M and all was well, free tells me the swap got to 107M.

Dare I suggest the swap file should be (yet another) option?

I don't think it needs to be an option because the whole point is to have it there if the OS needs it. It looks like it defaults to 200MB for RAM <= 4GB and 100MB for the 8GB Pi's. This should cover the moode usage scenarios where swap memory might be needed.

That locale error is odd though.
Did you set it using Pi Imager, raspy-config, other ?

I'll try to repro.