[PROBLEM] library update in progress

[arick]

Understand you.
I can send SD image, but I can't send music source ).
So I can dismount music source and try to reproduce the problem without it. I think the information about library updating state is stored on SD while the system is rebooting.
I try it two days later, the library is regenerated now.
Thank you

[TheOldPresbyope]

I have the same log

20200713 010030 worker: Job update_library
20200713 010031 thmcache: Start
20200713 010031 thmcache: Scanning: NAS, SDCARD, UPNP
   some messages about incorrect ID3v2 tags
20200713 175750 thmcache: Done: 28917 album dirs processed
20200713 232509 worker: Job update_library done

But I can't start update again

As Tim said, the thmcache and MPD update processes run in parallel. So, in the above, thmcache took nearly 17 hours to complete and MPD took about 22-1/2 hours.

These numbers make me suspicious of your "music library on PC". What exactly is this? OS, server, protocol? For that matter what are the different track file formats involved?

Here's a quick test from the command line of the communications with your library. This test runs the first step of moOde 6.6.0's thumbnail caching process, simply scanning all the directories of moOde's known music sources, looking for music tracks.

Code:

time /var/www/command/listall.sh > /dev/null

For the player I cited earlier, named moodeLR, it took ~5.5 seconds to find the 6980 tracks in 600+ directories.

Code:

pi@moodeLR:~ $ time /var/www/command/listall.sh>/dev/null

real    0m5.473s
user    0m0.224s
sys    0m0.328s

You can run this test even now. It doesn't force an update or regeneration.

Regards,
Kent

[arick]

The library placed on two PCs (three HD 3,5”) working under W7 and shared via Samba. 90% tracks are flac and mp3. Your script worked about 1 hour without results, till I have ended it by ^C

[TheOldPresbyope]

@arick

I don't know that your library server arrangement is the only problem but this appears to be very poor performance.

For the player I used in my previous examples, the track discovery rate was 6980 tracks/5.5 seconds = 4.5 million tracks/hour. My back-of-the-envelope estimate suggests your process should have been done in something like 5 minutes instead of more than the hour you waited before killing it.

One way to check just how fast listall.sh is detecting tracks in your library is to run a modified version of my test for a measured length of time, say 10 minutes or half an hour:

Code:

/var/www/command/listall.sh > /tmp/listall.out

At the end of the time interval, kill the process with CTRL-C and then take the number of files detected, which will be the number in the first column of output from the following command, and divide the number by the elapsed time.

Code:

wc /tmp/listall.out

Using my player again (for which, remember, listall.sh ran to completion in 5.5 seconds so I didn't need to CTRL-C):

Code:

pi@moodeLR:~ $ wc /tmp/listall.out
 6980 110500 955088 /tmp/listall.out

So, 6980 tracks and 5.5 seconds.

The number in the third column is interesting as well. That's the size of the list in bytes. In normal operation, that list is being accumulated in RAM and then sorted before being passed on to the thumbnail generator. A back-of-the-envelope calculation suggests your output might be around 50MB, and the sort operation will take space too. This could be a resource issue on an RPi model with, say 512MB total RAM, but I'm not sure it would be a problem on a 1GB RPi3B and surely not on a 2GB or 4GB RPi4B.

Let's see what you are actually getting. Whatever the result is, the process should have completed.

I think it possible that you are running into resource issues as Tim suggests might be happening, just because of the large size of your library, but I think it's also likely from the numbers you've posted so far that your library arrangement is inadequate.

Regards,
Kent

PS - My numbers were taken on an RPi4B. The rate would have been some 20 percent - 30 percent lower on an RPi3B.

[arick]

Dear Tim,
As I thought, you can reproduce the situation easy without music source. All that you have to own is RPI 3b+. I have send you the link to SD image in private. Just try to update the library or any folder

Dear Kent,
my results is
112084 1576425 13209600 tmp/listall.out

Regards,
Arkady

[TheOldPresbyope]

Dear Tim,
As I thought, you can reproduce the situation easy without music source. All that you have to own is RPI 3b+. I have send you the link to SD image in private. Just try to update the library or any folder

Dear Kent,
my results is
 112084  1576425 13209600 tmp/listall.out

Regards,
Arkady

I'll wait to hear from Tim what he makes of your image.

Meanwhile, your results show listall.sh scanned 16 times as many tracks while creating a file ca 14 times as big. The numbers seem quite reasonable. However, you neglected to say how long you let the listall.sh process run.

Regards,
Kent

[arick]

@TheOldPresbyope
Maybe 10 or 20 minites

[Tim Curtis]

I don't debug user images which may be corrupt or hacked.

[arick]

I don't debug user images which may be corrupt or hacked.

Why do you think that the image is corrupted or hacked? It's clear as a mountain river or Evian
By the other side it is no reason to solve the problem of the single user

[Tim Curtis]

I only debug stock moOde and I've already mentioned several times that I'm not able to repro your issue on my end so I'll be leaving this thread permanently.