[SOLVED] nginx Bad Gateway after idle

[TheOldPresbyope]

...

@TheOldPresbyope
Thanks for the pointer to the nginx log, i get a lot of this:

Code:

2023/08/23 09:51:34 [error] 546#546: *84994 connect() to unix:/run/php/php7.4-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 192.168.0.207, server: , request: "GET /engine-mpd.php?state=reconnect&_=1692777094581 HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock:", host: "192.168.0.123", referrer: "http://192.168.0.123/"

Not sure what could be using up resources, this pi is doing nothing but moode ¯\_(ツ)_/¯
I'll google around for that error a bit, but maybe someone has experienced something similar already.

This is not inconsistent with my conjecture. This is the overarching "FastCGI Process Manager". Looking at your original post, I see

Code:

  ~                                      ✘ INT  23s rpi@rpi  10:45:42 AM
╰─❯ systemctl status php7.4-fpm
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor pr>
   Active: active (running) since Wed 2023-08-09 10:06:34 CEST; 24h ago
     Docs: man:php-fpm7.4(8)
 Main PID: 497 (php-fpm7.4)
   Status: "Processes active: 64, idle: 0, Requests: 1852, slow: 0, Traffic: >
    Tasks: 65 (limit: 2137)
      CPU: 4min 4.210s
   CGroup: /system.slice/php7.4-fpm.service
           ├─  423 php-fpm: pool www
           ├─  437 php-fpm: pool www
           ├─  497 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
           ├─  568 php-fpm: pool www
    ...

When I look at moOde 8.3.5 running on an RPi4B here, I see this service is managing far fewer processes

Code:

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
    Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor pr>
    Active: active (running) since Tue 2023-08-22 17:32:21 EDT; 21h ago
      Docs: man:php-fpm7.4(8)
  Main PID: 467 (php-fpm7.4)
    Status: "Processes active: 13, idle: 3, Requests: 250, slow: 0, Traffic: 0>
     Tasks: 17 (limit: 2137)
       CPU: 6.209s
    CGroup: /system.slice/php7.4-fpm.service
            ├─   467 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
            ├─   546 php-fpm: pool www
            ├─   547 php-fpm: pool www
            ├─  3999 php-fpm: pool www
    ...

I wonder why you have 5 times as many processes running (even if this is well below the 2137 limit)?

Regards,
Kent

[demian.smith]

I know, necromancing is not en vogue - but the issue is back.

Exactly the same as before, including the startup user "pi" and the local user "rpi" and multiple threads for php...ssh works as usual, but the gui gives the "502 Bad Gateway" :/

Code:

hp7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
    Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
    Active: active (running) since Tue 2023-11-21 12:58:56 CET; 1 day 20h ago
      Docs: man:php-fpm7.4(8)
  Main PID: 496 (php-fpm7.4)
    Status: "Processes active: 64, idle: 0, Requests: 3792, slow: 0, Traffic: 0req/sec"
     Tasks: 65 (limit: 2137)
       CPU: 3min 42.744s
    CGroup: /system.slice/php7.4-fpm.service
            ├─   496 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
            ├─137203 php-fpm: pool www
            ├─192733 php-fpm: pool www
            ├─193315 php-fpm: pool www
            ├─194517 php-fpm: pool www
            ├─194691 php-fpm: pool www
            ├─194809 php-fpm: pool www
            ├─194835 php-fpm: pool www
            ├─195097 php-fpm: pool www
            ├─195127 php-fpm: pool www
            ├─195256 php-fpm: pool www
            ├─195328 php-fpm: pool www
            ├─195449 php-fpm: pool www
            ├─195458 php-fpm: pool www
            ├─195572 php-fpm: pool www

Code:

20231121 125913 worker: Host      (rpi)
20231121 125913 worker: Hardware  (Pi-4B 1.2 2GB)
20231121 125913 worker: moOde     (8.3.6 2023-09-05)
20231121 125913 worker: RaspiOS   (11.7)
20231121 125913 worker: Kernel    (6.1.21-v8+ #1642)
20231121 125913 worker: Procarch  (aarch64, 64-bit)
20231121 125913 worker: MPD ver   (0.23.12)
20231121 125913 worker: CPU gov   (ondemand)
20231121 125913 worker: Userid    (pi)
20231121 125913 worker: Homedir   (/home/pi)

The "weird" thing with php is that before the reboot systemctl gets "stuck", after the reboot I get

Code:

❯ systemctl status php7.4-fpm
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
    Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
    Active: active (running) since Thu 2023-11-23 09:38:06 CET; 12min ago
      Docs: man:php-fpm7.4(8)
  Main PID: 485 (php-fpm7.4)
    Status: "Processes active: 7, idle: 1, Requests: 65, slow: 0, Traffic: 0req/sec"
     Tasks: 9 (limit: 2137)
       CPU: 2.186s
    CGroup: /system.slice/php7.4-fpm.service
            ├─ 485 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
            ├─ 556 php-fpm: pool www
            ├─ 557 php-fpm: pool www
            ├─1474 php-fpm: pool www
            ├─1499 php-fpm: pool www
            ├─1504 php-fpm: pool www
            ├─1687 php-fpm: pool www
            ├─1747 php-fpm: pool www
            └─1752 php-fpm: pool www

Nov 23 09:38:05 rpi systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Nov 23 09:38:06 rpi systemd[1]: Started The PHP 7.4 FastCGI Process Manager.

This is a bog standard install, only changed the hostname via gui and set a static IP. I have 38379 tracks, so moode should be able to handle the e collection based on what I read here in the forum.The external disk does pass smartctl, so there's no problem there I believe.

Maybe I have to replace the SD Card?

[Nutul]

So you say that, since the end of august, the system has been running fine with no issues; and now suddenly the problem appeared again?

[demian.smith]

It started a few weeks back again, but was too busy to ask again and just kept rebooting the pi vie ssh instead...

[Nutul]

It started a few weeks back again, but was too busy to ask again and just kept rebooting the pi vie ssh instead...

And you did absolutely nothing to the installation. Added software, altered some configuration, maybe restored some backups, whatever?
It's extremely weird...
How does the free space on the SD card?

[demian.smith]

Nope, nothing, installed from a new downloaded dmg, re-configured everything (including the hostname) and let it run. Worked for a bit, then *poof*, issue started again.
The SD card space looks good:

Code:

❯ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  2.8G   11G  21% /
devtmpfs        891M     0  891M   0% /dev
tmpfs           925M     0  925M   0% /dev/shm
tmpfs           370M  1.3M  369M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p1  253M   32M  221M  13% /boot

[the_bertrum]

Have you tried a different SD card?

[fdealexa]

Hi demian.smith,
I have some doubts about the network:
- have you tried to connect the rpi via cable (ethernet)?
- have you any time limitation on the router or on the access point?
Best regards,
Francesco

[TheOldPresbyope]

...

This is a bog standard install, only changed the hostname via gui and set a static IP. I have 38379 tracks, so moode should be able to handle the e collection based on what I read here in the forum.The external disk does pass smartctl, so there's no problem there I believe.

Maybe I have to replace the SD Card?

Well, maybe, and it's easy enough to try another uSD card, but I'm wondering about your external disk and the music library on it. Have you tried your new installation with a smaller collection on, say, a USB memory stick? Does the 502 Bad Gateway problem still occur then?

A fundamental problem for us following your thread is that 1) none of us is a nginx/php expert (so can diagnose from first principles) and 2) none of us has been able to repro the situation based on what you report (so can diagnose based on experiences).

As an aside, I don't see any description in this thread of your physical setup. What is this external disk, how is it powered, and how is it connected (e.g., direct USB, USB hub)? You showed us the output from dh -h but apparently the external disk was not connected and its filesystem mounted. I'm particularly fond of the output from lsblk -f

You say the drive passes smartctl but that's a very low-level tool. What about filesystem-level checks? Also, have you looked at the MPD log to see if certain tracks/directories are giving it problems?

As you can see, I'm just blinding swinging at the piñata here but sometimes even a blind man manages to hit it;-)


Regards,
Kent

[demian.smith]

Hi all,
thanks for the input!
There's no limitation on the AP and the pi stands 1metre away from a laptop with good connection, Signal-to.noise and signal strength also look good. Additionally, i don't think the issue would happen after only idle if it were the network (plus, ssh is working no problem).

The library is on a external HDD drive, usb powered from the pi, via a usb3 enclosure. I didn't include the external disc in the df output cause I thought it might be irrelevant.

In fairness, I totally expect the SD card to just be fried, so I have been waiting for a new one which I have flashed just today, currently the database is rebuilding.

There's also one minor detail ijust relaised yesterday - the setup instructions state "Do not set the host name using the Imager" and I believe I might actually have done that, so maybe theat's the reason.

I'll just monitor it for now with the new 64 GB SanDisk microSDXC