Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Dmytro]

Apologies if off topic, as my library is rather small comparably (about 35,000 tracks flac). Every time I cold-open a client (i.e. after some since last run) MC would do a library load of about 60Mb from the server. This takes about 10-15 sec, so not a problem but a tad annoying.

Is there anything I can do to speed this up? How does it work for you with so much bigger libraries?

[RoderickGI]

Is there anything I can do to speed this up?

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

60MB sounds a bit big for 35,000 files. My 38,000 file Library is about 20MB. But I don't have a lot in the CD or Removed databases in that Library. You could clean out some of those older records if the time really bothers you, and you don't need the records in that part of the Library.

How does it work for you with so much bigger libraries?

I think they run large Libraries on a standalone PC, and not with Clients, like the OP. Or they just wait the time it takes to copy the Server Library to the Client.

[Dmytro]

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

Thanks Roderick, makes sense and is in the pipeline. Was just checking if there is anything unusual about such load volume for a library this size. There is not much deleted to justify the size, so nothing unusual then I guess.

[Dmytro]

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

60MB sounds a bit big for 35,000 files. My 38,000 file Library is about 20MB. But I don't have a lot in the CD or Removed databases in that Library. You could clean out some of those older records if the time really bothers you, and you don't need the records in that part of the Library.


I think they run large Libraries on a standalone PC, and not with Clients, like the OP. Or they just wait the time it takes to copy the Server Library to the Client.

Just realised the reason for why my library is bigger than I expect. Noticed that my library backups have gone up in size x10 times since January. Upon further inspection, it turns out that 54Mb out the current 65Mb (uncompressed) in my library is occupied by 'field (waveform).jmd' (I've re-run audio analysis recently for all the files). It practically does not get compressed in zip archive.

Just smth to be aware of.

[RoderickGI]

Ah. I didn't think of that.

I decided not to use the waveform display when it was implemented because of that issue. I want the MC Library to be small and fast, and I don't need the pretty display.

[mwillems]

Wow, thanks for running that down.  I've been experiencing a similar startup delay for a few months that I could not explain.  My library is currently 140MB, and 90MB of that is the waveforms!  I never put two and two together that my startup delays started around when I did the waveform analysis. 

I wonder if there are better compression options/techniques?  Or maybe a way to just store the information only in the files rather than in the library?  It's a neat feature, but it's not worth a 15-30 second wait for the wifi clients to startup.  I just tested with the waveforms removed and it cuts the delay down to five or ten seconds.

EDIT: RoderickGI, how did you disable MC from creating the waveform data during analysis?  I see a way to avoid saving the data in files, but not to avoid generating it in the library.

[JimH]

We'll find a fix.  Thanks for finding it.

[mwillems]

Thanks Jim!

[RoderickGI]

There is currently no way to stop the data being created. I just never turned on the display, and delete the field contents every now and then. To be honest though, I haven't imported a lot into my main Library on the HTPC for some time. I should check to see what is in there now... Nothing. I run a clean system! 

My test system does import a bunch of stuff, and the Waveform data is created.

I strongly advise turning off writing the values to tags in the files.

Previous discussions.
https://yabb.jriver.com/interact/index.php/topic,124225.0.html
https://yabb.jriver.com/interact/index.php/topic,124489.0.html
https://yabb.jriver.com/interact/index.php/topic,123053.msg860377.html#msg860377 Post includes data on how big Waveform field is.


That is good news Jim. Compress the data? Turn off the generation completely? Don't default it to being saved in files?

[DJLegba]

I routinely delete the waveforms after importing music. I would love to be able to tell MC not to run this analysis.

[mwillems]

That is good news Jim. Compress the data? Turn off the generation completely? Don't default it to being saved in files?

The first two would potentially help the issue in this thread, but the last one (changing the default to not save the info in the files) wouldn't help the delay issue, would it?  The large library size appears to be the cause of the startup delay, which is caused by all the waveform data being saved in the library as well as in the files.  If it were saved in the files only, instead of the library, that would reduce the library size/startup lag and potentially solve the issue in this thread.  Better compression, if possible, would also be a good solution.  And of course an option to turn it off would work, but I'd like to use it if I could do so in a performant way.

I get that some folks just want to be able to turn it off altogether, or are sensitive to the increase in disk space from writing the tags to the files, but I think the fixes for those issues may or may not be the same as the fix for this one?

[RoderickGI]

Well, the startup delay being discussed here is a result of a MC Client copying the Library from the MC Server. So a smaller Library should improve that, however you make the Library smaller.

If you use the Waveform, and store the information in the file tag but not in the Library, you would see other performance issues as MC would have to read from the file whenever that data was accessed. I guess if you are playing the file, that wouldn't be an issue. But if any View touched it, or the Tagging Window, then MC would have to wait while it read the tag from the file. I don't know if MC is even capable of only storing the data in the file tag, and not in the Library. I would think currently that it could not, so there could be development issues there.

The data appears to be plain text at the moment, so any compression would help, at the cost of processing power to compress and decompress it, which isn't much.

Having the data in each file, and hence having larger files, would only be an issue if the drive they were on was nearly full. Drives perform much worse when they are nearly full. But the data isn't that larger, so that is unlikely to be an issue.

The real answer for me would be to have the file analysed in real-time and displayed only if the feature was turned on. That would require more processing power in the Client, but maybe not much more. With that solution neither the Library nor the files would be larger. But issues such as seeking in a file would need to be addressed. The waveform may lag the audio a little, which probably wouldn't be appreciated. But that entirely depends on the implementation.

[Dmytro]

We'll find a fix.  Thanks for finding it.

Thanks Jim!

Well, the startup delay being discussed here is a result of a MC Client copying the Library from the MC Server. So a smaller Library should improve that, however you make the Library smaller.

Spot on, Roderick. Not as much of an issue for me, just a bit annoying. I can see it becoming a real issue for a larger library (such as @mwillems or bigger). Perhaps allowing to switch off waveform analysis can be a workaround, but does not seem to be a long-term solution.

This file is surprisingly hard to compress. ZIP (deflate: ultra) or 7ZIP (LZMA2: ultra) don't do anything at all to it.

[Matt]

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.

[mwillems]

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.

Matt, my "field (waveform).jmd" is 87.1MB.  My "field (filename).jmd" is 7.5MB.  So the waveforms are a bit more than ten times the size of the filenames for me.  The total size of my library directory is 139.4MB, the waveform.jmd is about 3/5 of the size of the total library.

About 110k total library files ~69k music ~9K video and the rest are images.

Thanks for looking into it.

[Matt]

It's just a lot of data.

We've got an idea to use a little fancier compression we might try eventually.

I just made it not output the leading "0." but instead just ".".

We toyed with not including this field in a library backup going to a client, but you might want it, so that's not the best.

Thanks again for your help.

[Hendrik]

This file is surprisingly hard to compress. ZIP (deflate: ultra) or 7ZIP (LZMA2: ultra) don't do anything at all to it.

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.
So we're still thinking about how to handle it.

[JimH]

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.
So we're still thinking about how to handle it.

How about breaking out one or more big pieces and having a two stage download, so the user can begin browsing while the second part is being retrieved?

[mwillems]

One thought I had was to treat the waveforms like you treat cover art or thumbnails.  Cover art isn't sent as part of the library send, the client just requests the art from the server as needed and then caches the thumbnails locally.  Waveforms could potentially work the same way with JIT delivery and local caching?

[zybex]

I just made it not output the leading "0." but instead just ".".

Why store this as text? Shouldn't it be a 16/32/64 bit float? Or the newfangled BFLOAT16, or even just integers, depending on the needed range.
Anyway, I agree with mwillems, JIT delivery + caching might be best for large data blobs.

[Dmytro]

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.

field (filename).jmd  - 621 KB
field (waveform).jmd - 52.9 Mb

37K audio files, very few video and images.

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.

Gotcha, this explains.

[RoderickGI]

Okay, here is something a little disturbing.

I had checked my HTPC audio files, and they had no Waveform data. I just check my file sizes and they are:

field (filename).jmd  - 2105 KB
field (waveform).jmd - 4025 KB

So I created a View that looked in all databases and found a lot of JTV, MKV, and MP4 files that had Waveform data in them. 334 files in total. They were imported from 24th March to 6th July 2020, the latest being TV recordings from last night in JTV format. The files were in a mixture of the Main and Removed databases.

Was it intended that video files get analysed for Waveform?

[JimH]

No, it's  not intended.  I thought we fixed that once.

[Matt]

Was it intended that video files get analysed for Waveform?

I was thinking about filtering out the waveform, but then I saw that we do display the waveform during video playback in Standard View and it works perfect.

[JimH]

I was thinking about filtering out the waveform, but then I saw that we do display the waveform during video playback in Standard View and it works perfect.

I don't think we should.

[Dmytro]

I don't think we should.

Just wanted to say huge thanks for fixing this in MC27! My library size is miraculously down to only 5MB and there is no noticeable start-up delay whatsoever.

Thank you!

[JimH]

Thanks for helping find it.

[RoderickGI]

I would still like to see the ability to turn Waveform off completely. It would be a good option to have.

I know a lot of people asked loudly for the waveform display, but the silent majority are now paying for it with either extra work to remove the data after Audio Analysis creates it, or a larger Library and slower startup times.