Upcoming moOde 8.2.1 release

[Tim Curtis]

Here are WIP release notes for upcoming 8.2.1 release (Q3 or Q4). This release provides some minor updates and a few bug fixes :-)

Code:

################################################################################
#
# 2022-MM-DD TC moOde 8.2.1
#
################################################################################

Updates

- UPD: Library "Recently added" is a persistent setting
- UPD: Add Artist (Strict) to Prefs Tag view artist options
- UPD: Improve layout and help text on Multiroom Config
- UPD: Set default for Receiver Master volume opt-in to Yes
- UPD: Remove ellipsis limit from CoverView wide layout

Bug fixes

- FIX: Margin for Volume popup close button (mobile)
- FIX: APD Router mode log entry in moOde startup log
- FIX: Maintenance interval 2 hours but should be 6 hours

[Tim Curtis]

Here are latest WIP release notes.

A mount monitoring option has been added to the File sharing feature. This replaces the basic mount monitor function in worker.php This is a dedicated daemon that automatically attempts to re-mount a Music source if it's mount point becomes unreachable. It supports both SMB and NFS mounts.

We are also looking at including the Airplay 2 capable version of shairport-sync if testing goes well.

Code:

################################################################################
#
# 2022-MM-DD TC moOde 8.2.1
#
################################################################################

Updates

- UPD: Bump to shairport-sync 4.1-dev with Airplay 2 support (tentative)
- UPD: Add mount monitoring option to File sharing feature
- UPD: Add Artist (Strict) to Prefs Tag view artist options
- UPD: Improve help text for File sharing
- UPD: Improve layout and help text on Multiroom Config
- UPD: Allow spaces in manually entered WiFi SSID
- UPD: Persist the Library "Recently added" setting
- UPD: Set default for Receiver Master volume opt-in to Yes
- UPD: Remove ellipsis limit from CoverView wide layout

Bug fixes

- FIX: Mediainfo run from PHP exec not handling accented file names
- FIX: Playlists not restored if backup has no radio stations
- FIX: Margin for Volume popup close button (mobile)
- FIX: APD Router mode log entry in moOde startup log
- FIX: Maintenance interval 2 hours but should be 6 hours

[suzywong]

A mount monitoring option has been added to the File sharing feature. This replaces the basic mount monitor function in worker.php This is a dedicated daemon that automatically attempts to re-mount a Music source if it's mount point becomes unreachable. It supports both SMB and NFS mounts.


That would be very useful, and for me timely!   Two of my streamers have, in the last month, lost their mount points (no apparent issues with eithernet or NAS). Different hardware configs & different versions of MoOde.

For sure, it's not been a critical issue, and a manual remount has got it back up, but in both cases I was left with an empty database, and had to do a "database update" to restore the library (is there a quicker way?), which obviously takes a bit of time but does however provide an opportunity to wander off the kitchen, make coffee, and stroke the kitties!

So t'would be nice to have an auto re-mount. Would (could) you auto trigger a database update as well?

[Tim Curtis]

lol, your cats trained you well ;-)

First I'd like to see how the improved mount monitor works in the field. It does the basics for example "is the host pingable" then if so "can the mount point be reached". The last one is where you can end up in the deep end of the pool with the Alligators :-0

[suzywong]

Fifteen years of Burmese cats has left us “on our knees”  .

Hear what you say, especially “it’s difficult to think straight when you are up to your a$$ in alligators”

[romain]

Here are latest WIP release notes.

A mount monitoring option has been added to the File sharing feature. This replaces the basic mount monitor function in worker.php This is a dedicated daemon that automatically attempts to re-mount a Music source if it's mount point becomes unreachable. It supports both SMB and NFS mounts.

Hello @Tim Curtis , 

nice to see a feature that I posted a while ago here: https://moodeaudio.org/forum/showthread.php?tid=2344 

The systemd service checks the availability of all sources, be it network (NFS, SMB, UPnP) or local (USB) sources, every 15 seconds and combines the auto re-mount function with leds.

I still use it with moOde 8.2.0, see files attached.

[Tim Curtis]

its a super nice script but the command (mountpoint) which is used in the script to check mount points will wait indefinitely if a mount point is an NFS mount and the NFS server service on the NAS has been turned off, is hung or has crashed. In this scenario the script will not continue checking mount points until the NFS server service comes back up on the NAS :-0

The approach I took in mountmon.php to check for NFS service availability is to probe port 2049. If this port is not open the script simply logs a warning and avoids checking the mount point which could result in an indefinite wait.

Code:

while (true) {
    session_id(phpSession('get_sessionid'));
    phpSession('open_ro');

    sleep(30);
    $mounts = sqlRead('cfg_source', $dbh);
    if ($mounts !== true) {
        mountmonLog('mountmon: Checking remote mounts');
        foreach ($mounts as $mp) {
            // See if host is up
            //mountmonLog('- Checking host ' . $mp['address'] . ' for mount ' . $mp['name']);
            $result = sysCmd('ping -A -4 -c 1 ' .  $mp['address'] . ' 2>&1 | grep "Destination Host Unreachable\|Name or service not known"');
            if (empty($result)) {
                mountmonLog('- Host ' . $mp['address'] . ' for mount point ' . $mp['name'] . ' appears to be up');
                // See if file sharing service is accessible on the host
                if ($mp['type'] == 'cifs') {
                    //mountmonLog('- Checking SMB mount ' . $mp['name']);
                    $result = sysCmd('ls /mnt/NAS/' . $mp['name'] .' 2>&1 | grep "Host is down\|Stale file handle"');
                    $fileSharingAccessible = !empty($result) ? false : true;
                }
                if ($mp['type'] == 'nfs') {
                    //mountmonLog('- Checking NFS mount ' . $mp['name']);
                    $port = '2049';
                    sysCmd('nmap -Pn -p ' . $port . ' ' . $mp['address'] . ' -oG /tmp/nmap.scan >/dev/null');
                    $result = sysCmd('cat /tmp/nmap.scan | grep "' . $port . '/open" | cut -f 1 | cut -d " " -f 2');
                    $fileSharingAccessible = !empty($result) ? true : false;
                }

                // Attempt remount
                if ($fileSharingAccessible === false) {
                    mountmonLog('- WARNING: Mount point ' . $mp['name'] . ' is unreachable');
                } else {
                    mountmonLog('- File sharing is accessible for ' . $mp['name']);
                    // Check for "Stale file handle" (NFS) or "Host is down" (SMB) return messages
                    // NOTE: This check can sometimes result in long timeouts or even a hang
                    //mountmonLog('- Checking ' . $mp['name'] . ' for stale file handle');
                    $result = sysCmd('ls /mnt/NAS/' . $mp['name'] . ' 2>&1 | grep "Host is down\|Stale file handle"');
                    if (!empty($result)) {
                        mountmonLog('- Attempting to re-mount ' . $mp['name'] . ' (stale file handle)');
                        sourceMount('unmountall');
                        sourceMount('mountall');
                    } else {
                        mountmonLog('- Mount ' . $mp['name'] . ' appears to be OK');
                    }
                }
            } else {
                mountmonLog('- WARNING: Host ' . $mp['address'] . ' for mount point ' . $mp['name'] . ' is unreachable');
            }
        }
    } else {
        mountmonLog('mountmon: No remote mounts are defined');
    }
}

  Here's some log output

Code:

pi@moode:~ $ tail -f /var/log/mountmon.log
20220930 075826 mountmon: Checking remote mounts
20220930 075826 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 075826 - File sharing is accessible for TRX-NFS-FLAC
20220930 075826 - Mount TRX-NFS-FLAC appears to be OK
20220930 075856 mountmon: Checking remote mounts
20220930 075856 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 075857 - File sharing is accessible for TRX-NFS-FLAC
20220930 075857 - Mount TRX-NFS-FLAC appears to be OK

# Turn NFS service off on NAS

20220930 075927 mountmon: Checking remote mounts
20220930 075927 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 075928 - WARNING: Mount point TRX-NFS-FLAC is unreachable
20220930 075958 mountmon: Checking remote mounts
20220930 075958 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 075958 - WARNING: Mount point TRX-NFS-FLAC is unreachable

# Turn NFS service back on

20220930 080028 mountmon: Checking remote mounts
20220930 080028 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 080029 - File sharing is accessible for TRX-NFS-FLAC
20220930 080029 - Mount TRX-NFS-FLAC appears to be OK
20220930 080059 mountmon: Checking remote mounts
20220930 080059 - Host 192.168.1.199 for mount point TRX-NFS-FLAC appears to be up
20220930 080059 - File sharing is accessible for TRX-NFS-FLAC
20220930 080059 - Mount TRX-NFS-FLAC appears to be OK

Also, a user posted an issue to our Git repo about a mount checking service built into systemd called Automount. I haven't tested it but apparently it depends on having the mounts in fstab. This would involve a rip and replace of how mounts are done in moOde and so its not something that has a high priority particularly if the basic mountmon daemon does a decent job.

[romain]

its a super nice script but the command (mountpoint) which is used in the script to check mount points will wait indefinitely if a mount point is an NFS mount and the NFS server service on the NAS has been turned off, is hung or has crashed. In this scenario the script will not continue checking mount points until the NFS server service comes back up on the NAS :-0

Tim, in my initial post, I had already raised the NFS problem, it's because NFS mounts are hard mounted by default. 
To avoid the script waiting indefinitely for NFS mounts or server to be back online, the solution is simply to use the soft mount option instead of the hard one.
The following NFS mount options work like a charm for me : soft,timeo=10,retrans=1,ro,nolock

[Tim Curtis]

Interesting. Similar to what is described in the article below?
https://kb.netapp.com/Advice_and_Trouble...soft_mount

I'll run some tests this weekend :-)

[romain]

Interesting. Similar to what is described in the article below?
https://kb.netapp.com/Advice_and_Trouble...soft_mount

Exactly.

IMHO, NFS soft mount is much more adapted than the hard one for a read only usage scenario as it is supposed to be the case with moOde.

Moreover, if I've well understood your approach, checking that the NFS server is online is not exactly the same thing as checking if NFS shares are online because testing the server port doesn't guarantee that NFS exports are available.