INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Startup Delay Across a Network  (Read 2691 times)

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Startup Delay Across a Network
« on: June 19, 2020, 08:10:38 pm »

Apologies if off topic, as my library is rather small comparably (about 35,000 tracks flac). Every time I cold-open a client (i.e. after some since last run) MC would do a library load of about 60Mb from the server. This takes about 10-15 sec, so not a problem but a tad annoying.

Is there anything I can do to speed this up? How does it work for you with so much bigger libraries?
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #1 on: June 19, 2020, 09:02:12 pm »

Is there anything I can do to speed this up?

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

60MB sounds a bit big for 35,000 files. My 38,000 file Library is about 20MB. But I don't have a lot in the CD or Removed databases in that Library. You could clean out some of those older records if the time really bothers you, and you don't need the records in that part of the Library.


How does it work for you with so much bigger libraries?

I think they run large Libraries on a standalone PC, and not with Clients, like the OP. Or they just wait the time it takes to copy the Server Library to the Client.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Re: Startup Delay Across a Network
« Reply #2 on: June 19, 2020, 09:22:30 pm »

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

Thanks Roderick, makes sense and is in the pipeline. Was just checking if there is anything unusual about such load volume for a library this size. There is not much deleted to justify the size, so nothing unusual then I guess.
Logged

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Re: Startup Delay Across a Network
« Reply #3 on: July 05, 2020, 01:21:20 am »

Get a faster network.
Get faster drives at each end.
Get a faster processor at each end.

60MB sounds a bit big for 35,000 files. My 38,000 file Library is about 20MB. But I don't have a lot in the CD or Removed databases in that Library. You could clean out some of those older records if the time really bothers you, and you don't need the records in that part of the Library.


I think they run large Libraries on a standalone PC, and not with Clients, like the OP. Or they just wait the time it takes to copy the Server Library to the Client.

Just realised the reason for why my library is bigger than I expect. Noticed that my library backups have gone up in size x10 times since January. Upon further inspection, it turns out that 54Mb out the current 65Mb (uncompressed) in my library is occupied by 'field (waveform).jmd' (I've re-run audio analysis recently for all the files). It practically does not get compressed in zip archive.

Just smth to be aware of.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #4 on: July 05, 2020, 02:35:48 am »

Ah. I didn't think of that.

I decided not to use the waveform display when it was implemented because of that issue. I want the MC Library to be small and fast, and I don't need the pretty display.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Startup Delay Across a Network
« Reply #5 on: July 05, 2020, 03:36:16 pm »

Wow, thanks for running that down.  I've been experiencing a similar startup delay for a few months that I could not explain.  My library is currently 140MB, and 90MB of that is the waveforms!  I never put two and two together that my startup delays started around when I did the waveform analysis. 

I wonder if there are better compression options/techniques?  Or maybe a way to just store the information only in the files rather than in the library?  It's a neat feature, but it's not worth a 15-30 second wait for the wifi clients to startup.  I just tested with the waveforms removed and it cuts the delay down to five or ten seconds.

EDIT: RoderickGI, how did you disable MC from creating the waveform data during analysis?  I see a way to avoid saving the data in files, but not to avoid generating it in the library.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72439
  • Where did I put my teeth?
Re: Startup Delay Across a Network
« Reply #6 on: July 05, 2020, 05:36:00 pm »

We'll find a fix.  Thanks for finding it.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Startup Delay Across a Network
« Reply #7 on: July 05, 2020, 06:23:39 pm »

Thanks Jim!
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #8 on: July 05, 2020, 07:22:38 pm »

There is currently no way to stop the data being created. I just never turned on the display, and delete the field contents every now and then. To be honest though, I haven't imported a lot into my main Library on the HTPC for some time. I should check to see what is in there now... Nothing. I run a clean system!  :D

My test system does import a bunch of stuff, and the Waveform data is created.

I strongly advise turning off writing the values to tags in the files.

Previous discussions.
https://yabb.jriver.com/interact/index.php/topic,124225.0.html
https://yabb.jriver.com/interact/index.php/topic,124489.0.html
https://yabb.jriver.com/interact/index.php/topic,123053.msg860377.html#msg860377 Post includes data on how big Waveform field is.


That is good news Jim. Compress the data? Turn off the generation completely? Don't default it to being saved in files?
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

DJLegba

  • Citizen of the Universe
  • *****
  • Posts: 995
Re: Startup Delay Across a Network
« Reply #9 on: July 05, 2020, 09:01:49 pm »

I routinely delete the waveforms after importing music. I would love to be able to tell MC not to run this analysis.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Startup Delay Across a Network
« Reply #10 on: July 05, 2020, 10:53:59 pm »

That is good news Jim. Compress the data? Turn off the generation completely? Don't default it to being saved in files?

The first two would potentially help the issue in this thread, but the last one (changing the default to not save the info in the files) wouldn't help the delay issue, would it?  The large library size appears to be the cause of the startup delay, which is caused by all the waveform data being saved in the library as well as in the files.  If it were saved in the files only, instead of the library, that would reduce the library size/startup lag and potentially solve the issue in this thread.  Better compression, if possible, would also be a good solution.  And of course an option to turn it off would work, but I'd like to use it if I could do so in a performant way.

I get that some folks just want to be able to turn it off altogether, or are sensitive to the increase in disk space from writing the tags to the files, but I think the fixes for those issues may or may not be the same as the fix for this one?
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #11 on: July 06, 2020, 12:56:44 am »

Well, the startup delay being discussed here is a result of a MC Client copying the Library from the MC Server. So a smaller Library should improve that, however you make the Library smaller.

If you use the Waveform, and store the information in the file tag but not in the Library, you would see other performance issues as MC would have to read from the file whenever that data was accessed. I guess if you are playing the file, that wouldn't be an issue. But if any View touched it, or the Tagging Window, then MC would have to wait while it read the tag from the file. I don't know if MC is even capable of only storing the data in the file tag, and not in the Library. I would think currently that it could not, so there could be development issues there.

The data appears to be plain text at the moment, so any compression would help, at the cost of processing power to compress and decompress it, which isn't much.

Having the data in each file, and hence having larger files, would only be an issue if the drive they were on was nearly full. Drives perform much worse when they are nearly full. But the data isn't that larger, so that is unlikely to be an issue.

The real answer for me would be to have the file analysed in real-time and displayed only if the feature was turned on. That would require more processing power in the Client, but maybe not much more. With that solution neither the Library nor the files would be larger. But issues such as seeking in a file would need to be addressed. The waveform may lag the audio a little, which probably wouldn't be appreciated. But that entirely depends on the implementation.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Re: Startup Delay Across a Network
« Reply #12 on: July 06, 2020, 02:45:34 am »

We'll find a fix.  Thanks for finding it.

Thanks Jim!

Well, the startup delay being discussed here is a result of a MC Client copying the Library from the MC Server. So a smaller Library should improve that, however you make the Library smaller.

Spot on, Roderick. Not as much of an issue for me, just a bit annoying. I can see it becoming a real issue for a larger library (such as @mwillems or bigger). Perhaps allowing to switch off waveform analysis can be a workaround, but does not seem to be a long-term solution.

This file is surprisingly hard to compress. ZIP (deflate: ultra) or 7ZIP (LZMA2: ultra) don't do anything at all to it.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42373
  • Shoes gone again!
Re: Startup Delay Across a Network
« Reply #13 on: July 06, 2020, 07:20:04 am »

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Startup Delay Across a Network
« Reply #14 on: July 06, 2020, 09:15:59 am »

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.

Matt, my "field (waveform).jmd" is 87.1MB.  My "field (filename).jmd" is 7.5MB.  So the waveforms are a bit more than ten times the size of the filenames for me.  The total size of my library directory is 139.4MB, the waveform.jmd is about 3/5 of the size of the total library.

About 110k total library files ~69k music ~9K video and the rest are images.

Thanks for looking into it.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42373
  • Shoes gone again!
Re: Startup Delay Across a Network
« Reply #15 on: July 06, 2020, 10:23:17 am »

It's just a lot of data.

We've got an idea to use a little fancier compression we might try eventually.

I just made it not output the leading "0." but instead just ".".

We toyed with not including this field in a library backup going to a client, but you might want it, so that's not the best.

Thanks again for your help.
Logged
Matt Ashland, JRiver Media Center

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10935
Re: Startup Delay Across a Network
« Reply #16 on: July 06, 2020, 10:30:37 am »

This file is surprisingly hard to compress. ZIP (deflate: ultra) or 7ZIP (LZMA2: ultra) don't do anything at all to it.

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.
So we're still thinking about how to handle it.
Logged
~ nevcairiel
~ Author of LAV Filters

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72439
  • Where did I put my teeth?
Re: Startup Delay Across a Network
« Reply #17 on: July 06, 2020, 10:39:04 am »

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.
So we're still thinking about how to handle it.
How about breaking out one or more big pieces and having a two stage download, so the user can begin browsing while the second part is being retrieved?
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Startup Delay Across a Network
« Reply #18 on: July 06, 2020, 10:51:42 am »

One thought I had was to treat the waveforms like you treat cover art or thumbnails.  Cover art isn't sent as part of the library send, the client just requests the art from the server as needed and then caches the thumbnails locally.  Waveforms could potentially work the same way with JIT delivery and local caching?
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Startup Delay Across a Network
« Reply #19 on: July 06, 2020, 11:27:53 am »

I just made it not output the leading "0." but instead just ".".

Why store this as text? Shouldn't it be a 16/32/64 bit float? Or the newfangled BFLOAT16, or even just integers, depending on the needed range.
Anyway, I agree with mwillems, JIT delivery + caching might be best for large data blobs.
Logged

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Re: Startup Delay Across a Network
« Reply #20 on: July 06, 2020, 06:13:28 pm »

Can you guys share the size of your "field (waveform).jmd" file in relation to your "field (filename).jmd" file?  In my library, I have the waveform totally filled and it's about half the size of my filename field.  So it's not going to be too terribly slow.

Let's start here, and see.  Thanks.

field (filename).jmd  - 621 KB
field (waveform).jmd - 52.9 Mb

37K audio files, very few video and images.

This is because its already compressed, but we'll try to use a better compression - but even if we manage to squeeze out another 50% or so (which I doubt), its still going to be a lot of data.

Gotcha, this explains.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #21 on: July 06, 2020, 07:24:20 pm »

Okay, here is something a little disturbing.

I had checked my HTPC audio files, and they had no Waveform data. I just check my file sizes and they are:

field (filename).jmd  - 2105 KB
field (waveform).jmd - 4025 KB

So I created a View that looked in all databases and found a lot of JTV, MKV, and MP4 files that had Waveform data in them. 334 files in total. They were imported from 24th March to 6th July 2020, the latest being TV recordings from last night in JTV format. The files were in a mixture of the Main and Removed databases.

Was it intended that video files get analysed for Waveform?

Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72439
  • Where did I put my teeth?
Re: Startup Delay Across a Network
« Reply #22 on: July 06, 2020, 09:32:17 pm »

No, it's  not intended.  I thought we fixed that once.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42373
  • Shoes gone again!
Re: Startup Delay Across a Network
« Reply #23 on: July 07, 2020, 08:26:57 am »

Was it intended that video files get analysed for Waveform?

I was thinking about filtering out the waveform, but then I saw that we do display the waveform during video playback in Standard View and it works perfect.
Logged
Matt Ashland, JRiver Media Center

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72439
  • Where did I put my teeth?
Re: Startup Delay Across a Network
« Reply #24 on: July 07, 2020, 08:46:02 am »

I was thinking about filtering out the waveform, but then I saw that we do display the waveform during video playback in Standard View and it works perfect.
I don't think we should.
Logged

Dmytro

  • Junior Woodchuck
  • **
  • Posts: 53
Re: Startup Delay Across a Network
« Reply #25 on: November 22, 2020, 12:51:52 am »

I don't think we should.

Just wanted to say huge thanks for fixing this in MC27! My library size is miraculously down to only 5MB and there is no noticeable start-up delay whatsoever.

Thank you!
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72439
  • Where did I put my teeth?
Re: Startup Delay Across a Network
« Reply #26 on: November 22, 2020, 06:12:30 am »

Thanks for helping find it.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Startup Delay Across a Network
« Reply #27 on: November 22, 2020, 04:27:45 pm »

I would still like to see the ability to turn Waveform off completely. It would be a good option to have.

I know a lot of people asked loudly for the waveform display, but the silent majority are now paying for it with either extra work to remove the data after Audio Analysis creates it, or a larger Library and slower startup times.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner
Pages: [1]   Go Up