[IDEA] Rendering very long lists

[moodenigo]

Hi

I have a very large library – tens of thousands of albums – and I am sure I am not the only one. In the Tag Library View, there is a noticeable lag when we have at least 10K elements in the scrollable list, and in my case it just gets stuck. See pic.

   

I do not know (yet) what causes this, but one possible culprit may be that this is a single list, which is rendered, and only a part of it is visible, but it is too large to properly be timely fetched.  I am trying to understand where this is in the codebase.

If this is the case, using some library (from react?) to render extremely long scrollable lists would be advisable. It would also speed up rendering for smaller libraries, making the UI even snappier.

Also, is the library data sent from the server to the client uncompressed or compressed?

[Tim Curtis]

What client are u using?

The data for Tag/Album views is generated in the loadLibrary() function in /var/www/inc/music-library.php. A file named libcache_all.json is returned to the client and stored in /var/local/www/. All data sent between client and server is automatically gzip compressed by NGINX, look in /etc/nginx/nginx.conf

There are some PHP, NGINX and MPD resource limits that cap the size of what can be transferred from server to client. 

PHP: Try doubling max_input_vars and post_max_size

Code:

   # /etc/php/$PHP_VER/fpm/php.ini
   # max_execution_time = 300
   # max_input_vars = 32768
   # memory_limit = -1
   # post_max_size = 75M
   # upload_max_filesize = 75M
   # session.save_path = "0;666;/var/local/php"

NGINX: Try doubling client_max_body_size

Code:

/etc/nginx/nginx.conf
# client_max_body_size 75M

MPD: Try doubling max_output_buffer_size

Code:

/etc/mpd.conf
# max_output_buffer_size "131072"

[moodenigo]

What client are u using?

You mean which browser? Safari on macOS.

[quote pid='55202' dateline='1718278376']
[b]Tim Curtis


[/b]The data for Tag/Album views is generated in the loadLibrary() function in /var/www/inc/music-library.php. A file named libcache_all.json is returned to the client and stored in /var/local/www/. All data sent between client and server is automatically gzip compressed by NGINX, look in /etc/nginx/nginx.conf

There are some PHP, NGINX and MPD resource limits that cap the size of what can be transferred from server to client. 

PHP: Try doubling max_input_vars and post_max_size

Code:

   # /etc/php/$PHP_VER/fpm/php.ini
   # max_execution_time = 300
   # max_input_vars = 32768
   # memory_limit = -1
   # post_max_size = 75M
   # upload_max_filesize = 75M
   # session.save_path = "0;666;/var/local/php"

NGINX: Try doubling client_max_body_size

Code:

/etc/nginx/nginx.conf
# client_max_body_size 75M

[/quote]

These make the UI a little more responsive, but it still lags. At least it refreshes.

[quote pid='55202' dateline='1718278376']

MPD: Try doubling max_output_buffer_size

Code:

/etc/mpd.conf
# max_output_buffer_size "131072"

[/quote]

In my case it is already 1048576, otherwise the library would not load at all.
I think once the data is there in libcache_all.json the list should be rendered in a more lazy way (if it is not already).
 Roberto

[moodenigo]

i think the problem is in the parsing of the libcache_all.json file. Even if rendering is lazy, parsing it takes time. A different format and library to read from it perhaps should be used.

[Tim Curtis]

lol, you can't just swap in a whole new library architecture. Ideally it would be an on-demand approach like Folder view uses but converting Tag/Album view to that type of architecture would be a huge undertaking.

The current architecture where the entire Library is loaded into the client has worked really well for most users, but it doesn't scale for ginormous sized collections like yours.

[moodenigo]

lol, you can't just swap in a whole new library architecture. Ideally it would be an on-demand approach like Folder view uses but converting Tag/Album view to that type of architecture would be a huge undertaking.

The current architecture where the entire Library is loaded into the client has worked really well for most users, but it doesn't scale for ginormous sized collections like yours.

I know, that it would not be easy. The fact that the library is served from the server to the client is ok.
A question is whether the library is loaded into the *memory* of the client, or whether the client caches a copy and parses it on demand?

[Tim Curtis]

lol, you can't just swap in a whole new library architecture. Ideally it would be an on-demand approach like Folder view uses but converting Tag/Album view to that type of architecture would be a huge undertaking.

The current architecture where the entire Library is loaded into the client has worked really well for most users, but it doesn't scale for ginormous sized collections like yours.

I know, that it would not be easy. The fact that the library is served from the server to the client is ok.
A question is whether the library is loaded into the *memory* of the client, or whether the client caches a copy and parses it on demand?

See /var/www/js/scripts-library.js

The Genre, Artist and Album lists are parsed from libcache_all.json and rendered in the WebUI. The track list is generated and rendered on-demand when Artist and/or Album is selected.

[swizzle]

lol, you can't just swap in a whole new library architecture. Ideally it would be an on-demand approach like Folder view uses but converting Tag/Album view to that type of architecture would be a huge undertaking.

The current architecture where the entire Library is loaded into the client has worked really well for most users, but it doesn't scale for ginormous sized collections like yours.

I know, that it would not be easy. The fact that the library is served from the server to the client is ok.
A question is whether the library is loaded into the *memory* of the client, or whether the client caches a copy and parses it on demand?

Ideally lists would be lazily loaded to get around memory constraints but like Tim said it’s a lot of work, which nobody is volunteering to do (and would likely kill some features). I don’t have time to do much on moode these days so I couldn’t say if this is just browsers hating big lists or if there is some pain point that could be improved short of throwing a complete library switcheroo at it. I did spend quite a few hours back in the day profiling JavaScript to improve list perf but that was years ago so I’m not sure what all has changed.

Putting aside that you can check to make sure thumbnails are disabled, ellipsis text is turned off and any hd, etc. labels are turned off. If that doesn’t help and your library is well maintained you can use genre or search to limit the number of shown artists/albums.

[Nutul]

I remember, few years ago, working on a multimedia console for the Mazda3 model year 2018... dealing with long lists of metadata.
We ended up creating a brand-new javascript LIST component, which actually loaded the whole library content in chunks, while you were scrolling it with your finger (discarding "older" content and requesting "newer" one).

We succeeded, of course; but It was not an easy task implementing it.

It can, of course, be done for mo\ode as well, but then again, re-read the above line...
Just my 2c.