Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[00formulahawk]

I've noticed that in MC25, when I add new movies to my library folder and run an auto-import (which also runs an audio analysis of the new files), it takes considerably longer than it does in MC24.

I tested auto-import on 3 specific videos last night in the following way:
- copy the files to my videos folder
- run auto-import on MC24
- move the files out of the videos folder to a temp folder
- delete the xml file for each of the videos
- move the videos back into the videos folder
- run auto-import on MC25

MC24 completes the test in 2:30-3:00 every time
MC25 completes the test in 8:30-9:00 every time

I then tested the audio analysis in the following way in both MC24 and MC25:
- select the same 3 video files from the previous test
- right click -> analyze audio
- uncheck the box to skip analyzed files
- process 1 file at a time
- repeated the same steps and processed 3 files at a time

Both MC24 and MC25 finished the analysis 1 file at a time in 2:30-3:00
Both MC24 and MC25 finished the analysis of all 3 at once in 8:30 to 9:00


I believe this is the feature that's causing the problem with auto-import:
     25.0.16 (3/25/2019)
          6. NEW: Made audio analysis done by auto-import threaded.

I've got an old 5400 RPM 1TB drive that my music and video library are stored on.  Multi-threaded audio analysis works great on smaller music files (~10MB) where it can load the entire file into memory quickly, but on video files, you're trying to simultaneously read multiple multi-gig files at the same time, and the seek speed of the drive is slowing the analysis down more than the extra processing power to analyze the data is helping.

Is there a way to disable this or specify how many threads it can use for the audio analysis during an auto-import?

[RD James]

I checked this, and can confirm that it's trying to analyze multiple videos at once during auto-import now, which is likely to destroy performance on HDDs.
I have said this many times, but I would like the "automatic" setting for analysis to be more context-aware.

Standard audio tracks should analyze as many files as your CPU has threads available.
Audio files are typically small enough that even loading these from an HDD is not going to hurt performance.
This is currently limited to 8. My CPU has 16 threads, and I plan on upgrading to 32 threads once the new Ryzen CPUs are available later this year.

DSD/SACD audio should analyze 1/4 as many tracks as your CPU has threads available.
DSD decoding uses 4 threads per track in Media Center, so this maximizes CPU utilization while minimizing disk access.
DSD tracks tend to be quite a bit larger than standard audio tracks, so this can noticeably improve the performance of analysis if stored on an HDD.

Video is tricky, since the CPU utilization can vary a lot depending on the audio formats used.
Analyzing multiple videos at once can cause significant disk thrashing and performance degradation if stored on a hard drive, sometimes slowing down the process considerably.
Unless you are dealing with very small files, it is typically best to analyze videos one at a time if they are stored on an HDD.
But if the videos are on faster storage, or stored across multiple drives, analyzing multiple files at once can speed up the process.

I ran a short test on 8x10GB videos, and analyzing four files at once on an NVMe SSD seemed optimal.
One file at a time took ~4 minutes, two files halved that, and four files halved it again - bringing it down to only 1 minute.
Analyzing 8 files at once only improved speed by another 20% or so, because I was then limited by the CPU rather than the disk.
But analyzing that many files at once is not good when they're stored on HDDs rather than an NVMe SSD. SATA SSDs will bottleneck far sooner too, but that typically won't cause performance degradation, the analysis just won't get any faster as you add more files.
 

I don't know if Media Center is tracking any kind of performance metrics during analysis that might be able to solve this problem for videos.
I have a tiered storage setup where I have SSDs for fast writes, files are then moved to HDDs, and eventually to 'archive' HDDs (SMR drives that I won't be buying more of).

• If I analyze a single UHD movie, I get ~150MB/s off one of the HDDs.
• If I analyze two UHD movies which are stored on separate drives in the pool, analysis proceeds at ~300MB/s.
• But if I selected two UHD movies which are stored in different locations on the same disk, total performance dropped to ~40MB/s due to thrashing (and that's with a drive supporting NCQ).

If MC was tracking these metrics it could start by analyzing one file, add more until total I/O performance starts to drop, and back off once it does.
Alternatively, a much simpler solution would be something like a "maximum videos to analyze simultaneously during auto-import" option, even if that may limit performance.


Fortunately, it does appear that Media Center still runs analysis as a very low priority when it's in the background, so this only seems to be an issue for manually initiated 'auto-import' commands.

[Matt]

Coming next build:
Changed: When running audio analysis on import, it only uses one thread at a time for the videos in the import array.

[RoderickGI]

Oh darn.

I actually noticed an improvement in analysis speed on small files TV shows I imported. Short programs that were only 350 to 500 Mb or so. This was good.

Maybe the selection of single file analysis could have been based on size, and available memory?

Not a big issue though. I only noticed when I manually ran Auto Import really.

[Hendrik]

Maybe the selection of single file analysis could have been based on size, and available memory?

Neither of those factors really makes a difference. Video is problematic because it has to read vastly more data (ie. all the video data) to get to the audio. File size has no impact on that really, and memory is mostly inconsequential either way.