Multiroom receiver crashes

[abrakadabra2k]

Hello,
This is happening on a moode 9.1.3 (also in versions before) with Raspberry Pi4, connected via HDMI to a Denon receiver. Connection via Wifi (5Ghz, signal should be ok). 
I have 1x server, 3 receivers. The other 2 Receivers are working flawlessly. 
But the "HDMI one" always needs a multiroom-receiver restart to play music from the server after I select it in on the server web ui. 
After the reset, it is working just fine.

This is a log excerpt (debug mode on obviously - connection started at 162736  ):

Code:

20241102 162601 DEBUG: chkSQL(): type in ('nfs', 'cifs')
20241102 162631 DEBUG: chkSQL(): type in ('nfs', 'cifs')
20241102 162701 DEBUG: chkSQL(): type in ('nfs', 'cifs')
20241102 162731 DEBUG: chkSQL(): type in ('nfs', 'cifs')
20241102 162736 DEBUG: chkValue(): trx_control -rx
20241102 162736 DEBUG: chkValue(): -rx
20241102 162736 DEBUG: chkValue(): Value is blank
20241102 162736 DEBUG: chkValue(): trx_control -rx
20241102 162736 DEBUG: chkValue(): -rx
20241102 162736 DEBUG: chkValue(): Value is blank
20241102 162736 DEBUG: chkSQL(): param='sessionid'
20241102 162736 DEBUG: chkValue(): /var/www/util/trx-control.php
20241102 162736 DEBUG: chkValue(): -rx
20241102 162736 DEBUG: chkSQL(): param='rx_mastervol_opt_in'
20241102 162736 DEBUG: chkSQL(): param='volmute'
20241102 162736 DEBUG: chkSQL(): param='rx_host'
20241102 162736 DEBUG: phpSession(get_status): status=PHP_SESSION_ACTIVE
20241102 162736 DEBUG: chkSQL(): param='sessionid'
20241102 162736 DEBUG: chkValue(): /var/www/util/trx-control.php
20241102 162736 DEBUG: chkValue(): -rx
20241102 162736 DEBUG: chkSQL(): param='rx_mastervol_opt_in'
20241102 162736 DEBUG: chkSQL(): param='volmute'
20241102 162736 DEBUG: chkSQL(): param='rx_host'
20241102 162736 DEBUG: phpSession(get_status): status=PHP_SESSION_ACTIVE
20241102 162742 DEBUG: chkValue(): trx_control -rx On
20241102 162742 DEBUG: chkValue(): -rx
20241102 162742 DEBUG: chkValue(): On
20241102 162742 DEBUG: chkValue(): Value is blank
20241102 162742 DEBUG: chkSQL(): param='sessionid'
20241102 162742 DEBUG: chkValue(): /var/www/util/trx-control.php
20241102 162742 DEBUG: chkValue(): -rx
20241102 162742 DEBUG: chkValue(): On
20241102 162742 DEBUG: chkSQL(): param='multiroom_rx'
20241102 162742 DEBUG: chkSQL(): No WHERE clause
20241102 162743 DEBUG: chkSQL(): param='rxactive'
20241102 162745 watchdog: Started Multiroom receiver after crash detected
20241102 162746 DEBUG: chkSQL(): param='sessionid'
20241102 162746 DEBUG: chkValue(): /var/www/util/trx-control.php
20241102 162746 DEBUG: chkValue(): -rx
20241102 162746 DEBUG: chkValue(): On
20241102 162746 DEBUG: chkSQL(): param='multiroom_rx'
20241102 162746 DEBUG: chkSQL(): No WHERE clause
20241102 162753 watchdog: Started Multiroom receiver after crash detected

from here on, it will try to restart the receiver forever, but it will fail. Only manually restarting the receiver will recover the device.
This is reproducable.

[Tim Curtis]

Leave debug logging on and look for the launch string for the receiver. It will look something like below. If you don't see it then restart the receiver and should show up in the log.

Code:

trx-rx -d plughw:0,0 -h 239.0.0.1 -p 1350 -m 64 -j 64 -f 960 -R 45 -D /tmp/trx-rxpid  >/dev/null

Then run it from the cmd line to see if it prints out anything interesting when it crashes.

Code:

# So watchdog doesn't restart the receiver
sudo killall -s 9 watchdog.sh

# Kill the receiver
sudo killall trx-rx

# Run from cmd line (replace "-D /tmp/trx-rxpid  >/dev/null" with "-v 2")
sudo trx-rx -d plughw:0,0 -h 239.0.0.1 -p 1350 -m 64 -j 64 -f 960 -R 45 -v 2

# When done testing, reboot

[abrakadabra2k]

okay, running with:
sudo trx-rx -d default:vc4hdmi1 -h 239.0.0.1 -p 1350 -m 128 -j 64 -f 960 -R 45 -v 2

Output (removed most '.')

Code:

trx 0.6.0
(C) Copyright 2020 Mark Hills <mark@xwax.org>
(C) Copyright 2021 Tim Curtis <tim@moodeaudio.org>

###########.............................................Terminated

The "terminated" is appearing when I remove the device from the server's Rx clients (un-tick the box on the server). Is this intended? If yes, I'll let it run for some more time and I'll see what happens. So far I could not reproduce it that way

[Tim Curtis]

I was going to mention setting the "ALSA output mode" option on Receiver Config to "IEC958" but I see from the launch string that its been set (default:vc4hdmi1) but hdmi1 is the second port on the pi4. I think u need to use the 1st port (hdmi0) on the pi4 for audio, at least thats what I've always used.

Un-checking the receiver sets it on/off/disabled flag to off and then kills it. You would see "Terminated" in the debug logging. Watchdog only restarts the receiver if it flag = on and it has crashed.

[abrakadabra2k]

Thanks Tim, this lead me in the right direction: After a restart the Pi needed to "see" the HDMI cable conneted to an active device (probably some communication or signal level check?) . But the Denon was in standby. Even the web UI tells me when starting a radio "Output is set to HDMI but no audio device was detected on the HDMI port." After shutting down my AVR the pi sometimes complains about that, sometimes not.
I am pretty sure I did not have this problem on Moode 8. I searched the web and found that there have been lots of changes going on regarding the HDMI interface with bookworm...

Anyway, I can workaround this by enabling the "hdmi passthrough" option in my AVR. I dont like it as it increases the standby current (0.1 to 0.5W), but for the time being its fine...
Unless someone has a hint on how to configure the config.txt (or nowadays also cmdline.txt) so that moode will always send audio via hdmi?

[Tim Curtis]

Right. The legacy mode firmware videocore driver in Bullseye didn't care whether anything was actually connected to the HDMI port but in Bookworm it does. This is probably leading up to true HDMI hot plug capability or something like that.

HDMI audio format support in Bookworm is lightyears ahead of the legacy driver though :-)

[abrakadabra2k]

I could Not Stop thinking about this topic and adapted my little Denon Script: https://github.com/abrakadabra2k/DenonSe...estartTxRx

This works fine, but i still need to restart the Receiver (part of the Script). I am Wondering why the watchdog ist Not able to Bring Up the trx-rx

[abrakadabra2k]

@Tim Curtis  I tried the watchdog part regarding trx-rx in a minimal example, which fails as well:

Code:

moode@LivingMoode:/var/www/daemon $ cat test.sh
/var/www/util/trx-control.php -rx On
moode@LivingMoode:/var/www/daemon $ sudo ./test.sh
PHP Fatal error:  Uncaught Error: Call to undefined function getAlsaIEC958Device() in /var/www/inc/multiroom.php:43
Stack trace:
#0 /var/www/util/trx-control.php(84): startMultiroomReceiver()
#1 /var/www/util/trx-control.php(22): rxOnOff()
#2 {main}
  thrown in /var/www/inc/multiroom.php on line 43

Maybe some include problem?
Admittedly, I have no experience with php and its usage in a shell script

Quick update: The script is working on another machine with a hifiberry output.
I guess the getAlsaIEC958Device is used only with HDMI, the watchdog is probably not working with HDMI and IEC958?

[Tim Curtis]

Are you using the first HDMI port, the one nearest the power port on the Pi4?

The launch string for trx-rx should have -d default:vc4hdmi0 and the audio device connected to the HDMI port must be on.

[abrakadabra2k]

Yes, I switched to this port as you suggested.
This is the launch string obtained from debug log (as before, with other port):
trx-rx -d default:vc4hdmi0 -h 239.0.0.1 -p 1350 -m 128 -j 64 -f 960 -R 45 -D /tmp/trx-rxpid  >/dev/null

If I start my AVR I would expect the watchdog to be able to restart the trx-rx, but this error keeps on appearing - because of reasons written above I think:

Code:

20241104 122723 watchdog: Started Multiroom receiver after crash detected
20241104 122723 DEBUG: chkSQL(): param='sessionid'
20241104 122723 DEBUG: chkValue(): /var/www/util/trx-control.php
20241104 122723 DEBUG: chkValue(): -rx
20241104 122723 DEBUG: chkValue(): On
20241104 122723 DEBUG: chkSQL(): param='multiroom_rx'
20241104 122723 DEBUG: chkSQL(): No WHERE clause
20241104 122727 DEBUG: chkSQL(): type in ('nfs', 'cifs')
20241104 122730 watchdog: Started Multiroom receiver after crash detected
20241104 122731 DEBUG: chkSQL(): param='sessionid'
20241104 122731 DEBUG: chkValue(): /var/www/util/trx-control.php
20241104 122731 DEBUG: chkValue(): -rx
20241104 122731 DEBUG: chkValue(): On
20241104 122731 DEBUG: chkSQL(): param='multiroom_rx'
20241104 122731 DEBUG: chkSQL(): No WHERE clause

If I launch the trx-rx manually via the "launch string" above, music is played