[PROBLEM] Library update in progress message

[TheOldPresbyope]

I'm trying to stay focused on @alexarz's posts because he started this thread and added details first.

Toward that end, I should note that I was a bit vague earlier. Adding a new music source (disk, NAS, ...) to moOde works in two parts. There's MPD  (from the separate MPD project) and there's moOde code which uses MPD.

MPD scans the directory(s) of the music source looking for audio files it can work with (as determined by the various plugins available in the MPD build and potentially limited by .mpdignore files). From these files, MPD creates a custom database. The entries consist of paths to the audio files it identified and the metadata extracted from them In the process,

moOde code works with the contents of the MPD database and with the coverart images it extracts from the audio files and/or the music source directory(s). As an aside, this implies moOde can know only the contents of the metadata tags extracted by MPD.

---Or something like that. I'm sure others will jump in to correct anything I get wrong---

Ok, so this process can go wrong in several places. First, MPD may choke on files for various reasons, of which malformed metadata is only one. Second, moOde code can choke on the contents of the MPD database, notably because of specifics of the metadata.

I have begun trying to deal with these two problems separately. On the one hand, I've been putting together a primitive file validation script based on the Swiss-army knife known as ffprobe. I'll put it up in my github account soon. On the other, I'm trying to create test files to reproduce the problems we're seeing reported here. Haven't made as much progress on this part.

Keeps me out of bars!

Regards,
Kent

[alexarz]

Tomorrow I will try to identify the problematic folder or track using mpdignore, I will report the results

[Yawarakaimono]

Message #61 of this thread states that submitLibraryUpdate will reset GLOBAL.libLoading if all goes well. That is not the complete story.


Somewhere in the code is a case 'libupd_done' where the spinner is hidden and loadLibrary() is called, which again will set GLOBAL.libLoading = true


In the debugger-network-view is shown that the cmd load_library is still not completed.

(30 minutes now) Maybe it will eventually so.

So the library-update proper is ready in a reasonable time;
It is loadLibrary() that follows that again sets GLOBAL.libLoading = true.



What can I do to find out more, why it takes so long?

[Tim Curtis]

Message #61 of this thread states that submitLibraryUpdate will reset GLOBAL.libLoading if all goes well. That is not the complete story.


Somewhere in the code is a case 'libupd_done' where the spinner is hidden and loadLibrary() is called, which again will set GLOBAL.libLoading = true


In the debugger-network-view is shown that the cmd load_library is still not completed.

(30 minutes now) Maybe it will eventually so.

So the library-update proper is ready in a reasonable time;
It is loadLibrary() that follows that again sets GLOBAL.libLoading = true.



What can I do to find out more, why it takes so long?

Correct. The scripts-library.js function below is the second part of the "Update library" process. This part renders the PHP generated libcache.json file into Tag and Album views. The first part is to update MPD database and thumbnails.

Code:

function loadLibrary() {
   //console.log('loadLibrary(): loading=' + GLOBAL.libLoading, currentView);
   GLOBAL.libLoading = true;
   // DEBUG:
   //console.log('loadLibrary(): GLOBAL.libLoading: ' + GLOBAL.libLoading);

   // miscLibOptions[0] Include comment tag Yes/No
   // miscLibOptions[1] Album key: Album@Artist | Album@Artist@AlbumID | FolderPath | FolderPath@AlbumID
   // Note: AlbumID refers to the MUSICBRAINZ_ALBUMID tag
   miscLibOptions = getMiscLibOptions();

   // Convert to Yes/No
   // miscLibOptions[1] Indicates whether MBRZ albumid tag is included in the album key
   // miscLibOptions[2] Created to indicate whether FolderPath is used in the album key
   miscLibOptions[2] = miscLibOptions[1].indexOf('FolderPath') != -1 ? 'Yes' : 'No';
   miscLibOptions[1] = miscLibOptions[1].indexOf('AlbumID') != -1 ? 'Yes' : 'No';

    var loading = setTimeout(function(){
        if (currentView == 'tag' || currentView == 'album') {
            notify(NOTIFY_TITLE_INFO, 'library_loading');
        }
    }, 2000);

    $.getJSON('command/music-library.php?cmd=load_library', function(data) {
        clearTimeout(loading);
       $('#lib-content').show();
        renderLibrary(data);
        if (currentView == 'album' || currentView == 'tag') {
           setLibMenuAndHeader();
       }
       if (SESSION.json['lib_scope'] == 'recent') {
           $('.view-recents').click();
       }

       GLOBAL.libRendered = true;
       GLOBAL.libLoading = false;
       // DEBUG:
       //console.log('loadLibrary(): GLOBAL.libLoading: ' + GLOBAL.libLoading);
    });
}

The call chain is:
- send 'load_library' command to command/music-library.php
- music-library.php calls the loadLibrary() function in inc/music-library.php

This PHP loadLibrary() function reads through the MPD database and generates the moode libcache_ json file thats sent back to the loadLibrary() function in scripts-library.js

The PHP loadLibrary() process could take a while depending on how many tracks. Look at the file size of /var/local/www/libcache_all.json (ls -l /var/local/www)

Also the scripts-library.js loadLibrary() function could take a while to sort, group and render a really large libcache_all.json file (huge number of tracks).

[Yawarakaimono]

Message #61 of this thread states that submitLibraryUpdate will reset GLOBAL.libLoading if all goes well. That is not the complete story.


Somewhere in the code is a case 'libupd_done' where the spinner is hidden and loadLibrary() is called, which again will set GLOBAL.libLoading = true


In the debugger-network-view is shown that the cmd load_library is still not completed.

(30 minutes now) Maybe it will eventually so.

So the library-update proper is ready in a reasonable time;
It is loadLibrary() that follows that again sets GLOBAL.libLoading = true.



What can I do to find out more, why it takes so long?

Correct. The scripts-library.js function below is the second part of the "Update library" process. This part renders the PHP generated libcache.json file into Tag and Album views. The first part is to update MPD database and thumbnails.

Code:

function loadLibrary() {
   //console.log('loadLibrary(): loading=' + GLOBAL.libLoading, currentView);
   GLOBAL.libLoading = true;
   // DEBUG:
   //console.log('loadLibrary(): GLOBAL.libLoading: ' + GLOBAL.libLoading);

   // miscLibOptions[0] Include comment tag Yes/No
   // miscLibOptions[1] Album key: Album@Artist | Album@Artist@AlbumID | FolderPath | FolderPath@AlbumID
   // Note: AlbumID refers to the MUSICBRAINZ_ALBUMID tag
   miscLibOptions = getMiscLibOptions();

   // Convert to Yes/No
   // miscLibOptions[1] Indicates whether MBRZ albumid tag is included in the album key
   // miscLibOptions[2] Created to indicate whether FolderPath is used in the album key
   miscLibOptions[2] = miscLibOptions[1].indexOf('FolderPath') != -1 ? 'Yes' : 'No';
   miscLibOptions[1] = miscLibOptions[1].indexOf('AlbumID') != -1 ? 'Yes' : 'No';

    var loading = setTimeout(function(){
        if (currentView == 'tag' || currentView == 'album') {
            notify(NOTIFY_TITLE_INFO, 'library_loading');
        }
    }, 2000);

    $.getJSON('command/music-library.php?cmd=load_library', function(data) {
        clearTimeout(loading);
       $('#lib-content').show();
        renderLibrary(data);
        if (currentView == 'album' || currentView == 'tag') {
           setLibMenuAndHeader();
       }
       if (SESSION.json['lib_scope'] == 'recent') {
           $('.view-recents').click();
       }

       GLOBAL.libRendered = true;
       GLOBAL.libLoading = false;
       // DEBUG:
       //console.log('loadLibrary(): GLOBAL.libLoading: ' + GLOBAL.libLoading);
    });
}

The call chain is:
- send 'load_library' command to command/music-library.php
- music-library.php calls the loadLibrary() function in inc/music-library.php

This PHP loadLibrary() function reads through the MPD database and generates the moode libcache_ json file thats sent back to the loadLibrary() function in scripts-library.js

The PHP loadLibrary() process could take a while depending on how many tracks. Look at the file size of /var/local/www/libcache_all.json (ls -l /var/local/www)

Also the scripts-library.js loadLibrary() function could take a while to sort, group and render a really large libcache_all.json file (huge number of tracks).

Right, then for me there are no more exceptions that may disturb the library-update.

Why is the spinner shut down before the second part with loadLibrary() is finished?

[alexarz]

I found the problematic album, here is the link to it
https://disk.yandex.ru/d/8Hvwr6ZZrVpcpA

I left only one track in the folder, I think this will be enough for testing.

But, I got rid of error #62, but I did not solve the problem of updating the selected folder.
After regenerating the library, I waited for some time, when there was an access to the disk with the library, and the variables took the following value -

Code:

GLOBAL.libRendered;
true
GLOBAL.libLoading;
false

After that, I started updating the SDCARD, it completed (the spinner stopped spinning and disappeared), but the access to the disk did not stop even after 30 minutes, although I updated the SDCARD. The variables were still -

Code:

GLOBAL.libRendered;
false
GLOBAL.libLoading;
true

And a separate folder is no longer updated. I still have not waited for the disk access to stop.

There are 17313 tracks in my library.

And also - when regenerating the library in the java console I see this error

Code:

playerlib.js:706
Uncaught ReferenceError: loadLibrary is not defined
   at Object.success (playerlib.js:706:40)
   at u (jquery-1.8.2.js:974:30)
   at Object.fireWith [as resolveWith] (jquery-1.8.2.js:1082:7)
   at N (jquery-1.8.2.js:7788:14)
   at XMLHttpRequest.n (jquery-1.8.2.js:8500:8)

[Nutul]

I found the problematic album, here is the link to it
https://disk.yandex.ru/d/8Hvwr6ZZrVpcpA

1. The track is a WAV file, not supposed to be tagged.
2. Nevertheless, it has 2 tag sections: a id3v2.4.0 and a RIFF INFO (the RIFF INFO appears to be in some Windows codepage, while the id3v2 is to my eyes UTF8)

I would convert the whole album to FLAC, and tag it correctly with a VorbisComment; and see if the problems disappear.
It's always some mess with WAV files... I know somebody tags them, but I have been beaten on my fingers every time I tried to by my masters (so, TBH, I tried only twice... go figure the masochism...) so, to this day I DO NOT tag them.
Since we have a wonderful, lossless, free codec by the name of FLAC, let's get rid of all the WAVs in favor of it, and then correctly tag it.

Just my 2c, HTH

P.S.
I am getting a little sick of all these discussions going on for days, just to see in the end that the problem is 99% due to the presence of files not tagged properly.
We should have a go-through list, and point every "library does not end updating" thread to it; it could save us some precious time, to be dedicated to other, more relevant issues.

[TheOldPresbyope]

Al got there first.

The WAV audio component is valid (no error reported by ffprobe) but the metadata are a tangle what with having more than one tagging scheme present (WAV INFO chunk but also Id3). This makes it hard to know what is being used where in the various bits of code which make up the moOde functionality.

Also, I'm not sure that a consistent character encoding is being used but I'm not expert in handling what I assume is Cyrillic. Maybe what I'm seeing is an artifact of the two tagging schemes and how the software is dealing with them.

Like Al, I'm a big fan of FLAC. Clean, lossless, can be compressed, and only one place to stash the metadata we're interested in, the VORBIS_COMMENT block.

Sorry, I don't have time today to dig deep into the handling of this file.

Regards,
Kent

[Yawarakaimono]

P.S.
I am getting a little sick of all these discussions going on for days, just to see in the end that the problem is 99% due to the presence of files not tagged properly.
We should have a go-through list, and point every "library does not end updating" thread to it; it could save us some precious time, to be dedicated to other, more relevant issues.

The 'problem' is that it is not possible to update the library when some Javascript variable is 'true'. Users have no way to check its value, unless they can open a Javascript console. I have to dig my Windows laptop from the cupboard, because an iPad hasn't such a console. A spinner is shown when a library is being updated but it is not shown when the new database contents is being copied to the UI, during which the variable is also 'true'. This copying takes much longer than the library update. For more than 15k tracks at least an hour or two, three. 
This is not an example of good human interface design. The user must be able to see on forehand that an action can't be taken, instead of afterwards getting an alert 'You loser'.

[alexarz]

I wrote that I deleted this album from the library, the error from #62 disappeared, but the problem with updating a separate folder did not disappear. And I spent almost the whole day to identify this album. It would probably be better to simply ignore such albums when building a library. Isn't that logical?
I gave a link to this track just as an example. I don't want you to waste your time trying to find a mistake in it. But it would be great if such albums didn't cause difficulties in the future.