[SOLVED] Moode unreachable through web, but SSH works?

[buckaroo]

Hi All - I'm having a frequent issue where suddenly my Pi 3+ connected on wifi running moOde 8.1.2 will become unreachable through a web interface, giving me a "502 Bad Gateway" message (this happened in 7 as well). When this happens I can usually SSH into it just fine. I've tried getting to the IP on several different browsers on different machines, so I don't think it's an issue with cache, as is suggested on some sites concerning this error. Rebooting through SSH doesn't fix the problem. Sometimes connecting an ethernet cable will bring it back, but this is inconvenient for a number of reasons. The only thing that seems to bring it back reliably is to re-flash my SD card. Has anyone seen this behavior before? Is there a way to address it through SSH, so I don't have to re-flash the SD and re-index my 2 Tb of music?
Thanks!

[Tim Curtis]

The symptoms suggest a network issue, hardware or media issue, or something has crashed on the Pi.

How many tracks are in your collection?

When the symptom occurs again run the commands below to check on some of the core services.

Code:

# Look for log entries from watchdog indicating "Info: Reducing PHP fpm worker pool"
moodeutl -l

# MPD, PHP and NGINX status
systemctl status mpd
systemctl status nginx
systemctl status php7.4-fpm

# Network interface status
# Look for other than TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ifconfig

# Ping your router for 20 seconds and then examine the ping stats
# Look for long round trip times, dropped packets etc
# Ctrl-c stops the ping test
ping ROUTER_IP_OR_HOSTNAME

[buckaroo]

Thanks, Tim! I didn't see any log entries regarding PHP, but you might see something else here that I would miss:

Code:

20220921 001547 worker: --
20220921 001548 worker: -- Start moOde 8 series
20220921 001548 worker: --
20220921 001548 worker: Successfully daemonized
20220921 001552 worker: File check (OK)
20220921 001553 worker: Session vacuumed
20220921 001553 worker: Session loaded
20220921 001553 worker: Debug logging (OFF)
20220921 001553 worker: --
20220921 001553 worker: -- Audio debug
20220921 001553 worker: --
20220921 001553 worker: ALSA cards: (0:sndrpihifiberry | 1:empty | 2:empty | 3:empty
20220921 001553 worker: MPD config: (0:HiFiBerry DAC+ Pro | mixer:(Digital) | card:0)
20220921 001553 worker: ALSA mixer actual (Digital)
20220921 001553 worker: ALSA Digital volume set to (0%)
20220921 001553 worker: --
20220921 001553 worker: -- System
20220921 001553 worker: --
20220921 001556 worker: Host      (moode-br)
20220921 001556 worker: moOde     (8.1.2 2022-07-08)
20220921 001556 worker: RaspiOS   (11.3)
20220921 001556 worker: Kernel    (5.15.32-v8+ #1538)
20220921 001556 worker: Platform  (Pi-  )
20220921 001556 worker: ARM arch  (aarch64, 64-bit)
20220921 001556 worker: MPD ver   (0.23.5)
20220921 001556 worker: CPU gov   (ondemand)
20220921 001556 worker: USB boot  (not available)
20220921 001556 worker: File sys  (expanded)
20220921 001556 worker: HDMI port (On)
20220921 001557 worker: Sys LED0  (On)
20220921 001557 worker: Sys LED1  (On)
20220921 001557 worker: --
20220921 001557 worker: -- Network
20220921 001557 worker: --
20220921 001557 worker: eth0 adapter exists
20220921 001557 worker: eth0 check for address (No)
20220921 001557 worker: eth0 address not assigned
20220921 001557 worker: wlan0 adapter exists
20220921 001557 worker: wlan0 country (US)
20220921 001557 worker: wlan0 connected SSID is (Solaris)
20220921 001557 worker: IP addr (192.168.1.119)
20220921 001557 worker: Netmask (255.255.255.0)
20220921 001557 worker: Gateway (192.168.1.1)
20220921 001557 worker: Pri DNS (192.168.1.1)
20220921 001557 worker: Domain  (lan)
20220921 001558 worker: --
20220921 001558 worker: -- Audio config
20220921 001558 worker: --
20220921 001559 worker: MPD conf updated
20220921 001559 worker: ALSA card number (0)
20220921 001559 worker: MPD audio output (HiFiBerry DAC+ Pro)
20220921 001559 worker: Audio formats (S16_LE, S24_LE, S32_LE)
20220921 001559 worker: ALSA mixer name (Digital)
20220921 001559 worker: MPD mixer type (fixed 0dB)
20220921 001559 worker: Hdwr volume controller exists
20220921 001559 worker: Max ALSA volume (100%)
20220921 001559 worker: ALSA output mode (Default: plughw)
20220921 001559 worker: ALSA loopback (Off)
20220921 001559 worker: Reset renderer active flags
20220921 001559 worker: CamillaDSP (off)
20220921 001559 worker: --
20220921 001559 worker: -- MPD startup
20220921 001559 worker: --
20220921 001601 worker: MPD started
20220921 001601 worker: MPD accepting connections
20220921 001601 worker: MPD output 1 ALSA Default (On)
20220921 001601 worker: MPD output 2 ALSA Bluetooth (Off)
20220921 001601 worker: MPD output 3 HTTP Server (Off)
20220921 001601 worker: MPD crossfade (off)
20220921 001601 worker: MPD ignore CUE files (yes)
20220921 001601 worker: --
20220921 001601 worker: -- Music sources
20220921 001601 worker: --
20220921 001601 worker: USB sources (none attached)
20220921 001603 worker: NAS sources (mountall initiated)
20220921 001603 worker: --
20220921 001603 worker: -- Feature availability
20220921 001603 worker: --
20220921 001603 worker: Source select (available)
20220921 001603 worker: Source select (source: MPD)
20220921 001603 worker: Source select (output: HiFiBerry DAC+ Pro)
20220921 001603 worker: Bluetooth (available)
20220921 001603 worker: Airplay renderer (available)
20220921 001603 worker: Spotify renderer (available)
20220921 001603 worker: Squeezelite (available)
20220921 001603 worker: RoonBridge renderer (not installed)
20220921 001603 worker: Multiroom sender (available)
20220921 001603 worker: Multiroom receiver (available)
20220921 001603 worker: UPnP renderer (available)
20220921 001603 worker: DLNA server (available)
20220921 001603 worker: GPIO button handler (available)
20220921 001603 worker: Stream recorder (n/a)
20220921 001603 worker: --
20220921 001603 worker: -- Other
20220921 001603 worker: --
20220921 001603 worker: USB volume knob (Off)
20220921 001603 worker: USB auto-mounter (udisks-glue)
20220921 001603 worker: Saved MPD vol level (0)
20220921 001603 worker: Preamp volume level (0)
20220921 001603 worker: MPD volume level (0) restored
20220921 001603 worker: ALSA Digital volume (100%)
20220921 001603 worker: Auto-play (Off)
20220921 001603 worker: CoverView toggle (-off)
20220921 001603 worker: Maintenance interval (120 minutes)
20220921 001603 worker: Screen saver activation (Never)
20220921 001603 worker: Session permissions (OK)
20220921 001604 worker: Watchdog started
20220921 001604 worker: Ready

Mpd.service appears to be dead, for some reason:

Code:

● mpd.service - Music Player Daemon
    Loaded: loaded (/lib/systemd/system/mpd.service; disabled; vendor preset: enabled)
    Active: inactive (dead)
      Docs: man:mpd(1)
            man:mpd.conf(5)
            file:///usr/share/doc/mpd/html/user.html

Nothing looks unusual with nginx.service:

Code:

● nginx.service - A high performance web server and a reverse proxy server
    Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2022-10-01 14:47:25 EDT; 4h 8min ago
      Docs: man:nginx(8)
   Process: 432 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Process: 451 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Main PID: 452 (nginx)
     Tasks: 5 (limit: 1010)
       CPU: 174ms
    CGroup: /system.slice/nginx.service
            ├─452 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
            ├─453 nginx: worker process
            ├─454 nginx: worker process
            ├─455 nginx: worker process
            └─456 nginx: worker process

But something seems to be wrong with php7.4-fpm.service:

Code:

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
    Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
    Active: failed (Result: signal) since Sat 2022-10-01 14:47:24 EDT; 4h 11min ago
      Docs: man:php-fpm7.4(8)
   Process: 433 ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf (code=kil>
   Process: 438 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/poo>
  Main PID: 433 (code=killed, signal=SEGV)
       CPU: 41ms

Oct 01 14:47:24 moode-br systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Oct 01 14:47:24 moode-br systemd[1]: php7.4-fpm.service: Main process exited, code=killed, status=11/SEGV
Oct 01 14:47:24 moode-br systemd[1]: php7.4-fpm.service: Failed with result 'signal'.
Oct 01 14:47:24 moode-br systemd[1]: Failed to start The PHP 7.4 FastCGI Process Manager.

My longest ping was about 8 ms, but the vast majority were around 2-4 ms. Seems okay to me.
Also didn't see any other than 0 when I ran ifconfig.

So it looks like the culprit might lie in the MPD and/or PHP services - but I'm too much of a noob to know what to do about it. Any ideas?

Thanks!

[Tim Curtis]

How many tracks are in your collection?

MPD and PHP are dead which is completely abnormal.

This log entry is suspicious. It should identify your Pi model.
20220921 001556 worker: Platform (Pi- )

Maybe try a fresh image on a new SD Card.

[TheOldPresbyope]

SEGV signals are never good.

Code:

  Process: 433 ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf (code=kil>
  Process: 438 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/poo>
 Main PID: 433 (code=killed, signal=SEGV)

A faulty SD card or a corrupt image seem definite possibilities to me.

Regards,
Kent

[buckaroo]

Thanks, guys. Regarding the number of tracks, I couldn't tell you off the top of my head, but the thumbnail generator scanned 5368 folders, so the number is likely in the multiple tens of thousands. (Is there a way to find the actual number within moOde?)

I was hoping avoid another reflash, since it takes so long to set up and index my collection - but it sounds like that may be my only option. It probably *is* a questionable card - I thought I'd put a new one in, but that was in my other pi/moode system - the one I'm currently having trouble with is indeed a cheap Microcenter-branded no-name. I'll upgrade the card and see what happens.

[Tim Curtis]

Tag view shows the total number of tracks in the collection below the cover art in the lower right. The command "mpc stats" also shows number of tracks.

The number of folders would suggest maybe 50,000 tracks. I would think that with a locally attached disk it wouldn't take too long for MPD to index the tracks. IIRC the ALAC format is the slowest format to index so if you have a lot pf ALAC encoded tracks indexing is prolly going to take longer than if the tracks were FLAC.

[buckaroo]

Quick update: replaced the card, and so far, so good! But time will tell. Thanks for the advice!