Topic: What's likely to be the bottleneck in MC14 with a massive library? (Read 3706 times)

steveklein · « **on:** August 18, 2009, 10:58:33 am »

When you start talking about very large libraries of 100,000+ files and loading tons and tons and tons of thumbnails, my computer starts to kind of huff and puff a little. The processor is a Core 2 Duo at 2.13GHz and I've got 2 GB of 667MHz RAM on Windows 7 x86.

Any guesses as to what the bottleneck likely is? I'm kind of thinking throwing more RAM in there might increase performance. I just built a Core 2 Quad system with 8GB of 800MHz RAM that I'll be running Windows 7 x64 on. I'm very interested to see how the system performs.

Also, would storing most of my media (mp3s) on the same drive as the MC (and Win 7) see any performance increase than if I store them on a different drive?

JimH · « **Reply #1 on:** August 18, 2009, 11:17:33 am »

What is the library? Break down by media type?

Thumbnailing takes some time to complete if you have many images. You can force MC to build them all at once.

steveklein · « **Reply #2 on:** August 18, 2009, 12:13:29 pm »

70,000 mp3s, 5,000 jpg images that are covert art, an additional 5,000 jpgs that are from my camera, several hundred DVDs, over 2000 TV episodes, and I'm planning on starting to organize all my pdfs, MS Office files, and some relevant txt files which will be well intot he thousands.

if there is an ideal way to store the cover art, id like to know. i don't feel as if the way i've got it right now is all that good.

JimH · « **Reply #3 on:** August 18, 2009, 12:22:49 pm »

Thumbnailing is probably putting a load on your PC. Go to Tools/Options/Tree&View/Thumbnails and build them all at once.

A network drive could also be a bottleneck.

For performance, 4GB would be better.

steveklein · « **Reply #4 on:** August 18, 2009, 01:20:48 pm »

so a networked drive could, but as long as the files are in the same box as the installation of MC and Windows, but on separate physical drives, it should not negatively impact performance?

i will check that option to see what i have set when i get home

JimH · « **Reply #5 on:** August 18, 2009, 01:38:21 pm »

Separate internal drives would have no effect, assuming the drives are all the same speed.

krupted · « **Reply #6 on:** August 18, 2009, 04:11:53 pm »

i would have to agree, ram might be what op needs. 2gb is fine but its not optimal for windows longhorn

also, what motherboard do you have? some are better then others when it comes to how well data flows, especially in times where there are a lot of different files. i have had a few budget boards that are just 'clunkier' when it comes to dealing with these situations- even if they work great for games and whatever else.

Matt · « **Reply #7 on:** August 18, 2009, 04:20:52 pm »

Media Center likes multi-core CPUs. We've been working hard to parallelize everything we can these last years.

Normally memory usage is modest, even with large libraries, so more memory is less likely to help. Of course, if Windows itself is paging, it'll kill everything.

For thumbnail performance, having a fast Windows drive helps.

Keeping the library on a fast drive helps, but only with regards to load time. Once data is loaded, operations will be in memory.

Also, sometimes just a certain view is slow because a single overly complicated expression or pane is being used. More details about what causes the computer to "huff and puff" might help us get to the bottom of that.

datdude · « **Reply #8 on:** August 18, 2009, 06:54:09 pm »

What about in relation to theater view? I think I remember somewhere that it is more CPU intensive rather than GPU?

DrFloyd · « **Reply #9 on:** August 18, 2009, 07:10:13 pm »

I run on multiple systems, each with their own library.

Vista x86 on a Core 2 Duo 3.16GHz w/ 4GB RAM. 300,000+ Library on separate volume from OS drive and page file... Import of 300,000+ MP3's takes 30 minutes (with forced thumbnails) and start up is instant.

Older, AMD x2 Dual Core 4400 (2.21Ghz) w/ 4GB RAM. Running Windows 7 x64 RTM, has about the same performance. A little longer on the import, about 50 minutes (Slower drives). But start up is a little slower than instant.

Other features work better on my main, but not noticeably (drives are faster / better on the main box).

There shouldn't be anything in MC that's GPU intensive if you have a modern vid card. Notice no difference with MC between a 9800GX2; a 8800GTX; or a 7900GTX. Have a machine around here with a 6800GT, but that would run MC in Theater mode just fine as well.

globetrotters1 · « **Reply #10 on:** August 18, 2009, 08:50:43 pm »

Quote from: DrFloyd on August 18, 2009, 07:10:13 pm

Vista x86 on a Core 2 Duo 3.16GHz w/ 4GB RAM. 300,000+ Library on separate volume from OS drive and page file... Import of 300,000+ MP3's takes 30 minutes (with forced thumbnails) and start up is instant.

DrFloyd,

I just don't want to believe what you are saying... I have around 250'000 tracks (half MP3, half FLAC) on my disks right now and I need at least (!) one whole day to import into a new library - in packages of around 30'000 tracks each run, to avoid of the resulting library of being corrupt - and the startup is everything else than instant... something like 20 seconds!

Oh and I use a dual processor system with 2.8 GHz, 4GB RAM, a special fast OS drive and a very fast RAID 6 media storage system (ARECA controller)

So, I would be very grateful if you would list your hardware here for me, thanks

And for hardware hints of everyone else I'd be grateful too (as I want to switch over to a quad CPU M/B soon)

JimH · « **Reply #11 on:** August 18, 2009, 08:53:15 pm »

Quote from: datdude on August 18, 2009, 06:54:09 pm

What about in relation to theater view? I think I remember somewhere that it is more CPU intensive rather than GPU?

Should be graphics hardware dependent.

JimH · « **Reply #12 on:** August 18, 2009, 08:56:27 pm »

Quote from: globetrotters1 on August 18, 2009, 08:50:43 pm

I have around 250'000 tracks (half MP3, half FLAC) on my disks right now and I need at least (!) one whole day to import into a new library -

Could be a virus checker. Could be drive related (network, USB). If your library is corrupted, could be bad memory, bad DirectShow filter, etc. See the link called "Stability" in my signature for ideas.

You haven't said what version you run.

globetrotters1 · « **Reply #13 on:** September 07, 2009, 11:40:05 am »

Jim,

1) there is no virus scanner installed, as it is just an internal system without connection out of the home network
2) drives are connected with a Areca RAID controller - right now 10 500GB drives on RAID 6 - works extremely fast and without any flaw whatsoever
3) I re-built my whole library and it seems to be ok, no corruption message, nothing
4) I back up my library every day
5) memory is fine, I checked it and it shows zero errors, 4GB
6) DirectShow filters, there I have not the slightest idea, as I only rarely use TV and some DVD video (which is not sooo important for me), 99% the system is used for audio
7) I checked almost everything I can imagine, have no idea where I could check more

I installed now Win2003 Server SP2 on this system. Installation went fine. I installed MC13 and MC14 (test version). The import of my 250'000 tracks in MC14 needed something like 5 hours after I set the automatic import paths (manually with MC13 I needed a whole day). As I said I use 50% MP3 320, 50% FLAC and within the library like 20 special custom library fields (which might be the cause, who knows). I use a dual 2.8 GHz CPU and enough memory. And I use a blazingly fast hard disk storage (around 600MB/s sequential read, writes are close to that).

I had a problem with the video driver and the DirectX installation, so I re-installed and everything seems to work fine.

If you can tell me with which motherboard and quad CPU I can get a re-build of my library in 30 minutes, I'd be very happy to buy that...

I'm serious, so please let me know what the problem might be

gappie · « **Reply #14 on:** September 07, 2009, 11:47:17 am »

maybe the audio analyzing is on? guess that would take a while when importing so much.

gab

globetrotters1 · « **Reply #15 on:** September 07, 2009, 11:56:53 am »

Gappie,
sure it is on, but all tracks are already analyzed... I suppose then it doesn't do the whole analyzing stuff again...

hit_ny · « **Reply #16 on:** September 07, 2009, 02:19:17 pm »

Quote from: krupted on August 18, 2009, 04:11:53 pm

also, what motherboard do you have? some are better then others when it comes to how well data flows, especially in times where there are a lot of different files. i have had a few budget boards that are just 'clunkier' when it comes to dealing with these situations- even if they work great for games and whatever else.

So i guess not going for a budget board solves the problem ?

Any recommendations btw ?

globetrotters1 · « **Reply #17 on:** September 07, 2009, 03:02:57 pm »

or, Matt, why don't you tell us the specifications of your system you develop and test MC14 on?

this might be a useful information to make things easier and smoother for us

leezer3 · « **Reply #18 on:** September 07, 2009, 05:41:42 pm »

Something is definitely up with your system if it is importing that slowly.
My system (C2D 2.5ghz, 4gb RAM) normally takes around 1-2 mins to import 10,000 MP3s. Building thumbnails does slow things down somewhat, but nowhere near the levels you're suggesting.

Re. budget boards- Rubbish, you're imagining things. Assuming all the drivers are correctly up to date, and the two boards & specs are broadly similar then the performance will be as near to identical as it can be. The only differences I can think of that might be even remotely noticeable is if you're using a board which has bugged chipset drivers, and even then this can happen to both budget and expensive boards.
I suggest you read up on the placebo effect

-Leezer-

globetrotters1 · « **Reply #19 on:** September 08, 2009, 11:33:18 am »

leezer3,

I'm very sorry, and no insult intended, but i don't believe your numbers...

Our library now has approx 250'000 tracks on 3.2 terabyte of HD space - all MP3 320 or FLAC - analyzing activated, thumbnails activated - around 20 custom database fields, some very large - and i need full 5 hours of a complete new import!

My motherboard might be better, my CPU, memory and probably it's also the Win2003 server O/S, who knows

If someone can tell me what is wrong with my 5 hours or with my system then please don't hesitate to tell it

Matt · « **Reply #20 on:** September 08, 2009, 11:51:36 am »

Quote from: globetrotters1 on September 08, 2009, 11:33:18 am

leezer3,

I'm very sorry, and no insult intended, but i don't believe your numbers...

Our library now has approx 250'000 tracks on 3.2 terabyte of HD space - all MP3 320 or FLAC - analyzing activated, thumbnails activated - around 20 custom database fields, some very large - and i need full 5 hours of a complete new import!

My motherboard might be better, my CPU, memory and probably it's also the Win2003 server O/S, who knows

If someone can tell me what is wrong with my 5 hours or with my system then please don't hesitate to tell it

14 files/second doesn't seem that bad. It's going to depend on the speed of the drives and the file type (some format SDKs are faster than others). As a comparison, I'm getting about 60 files/second with APE files on a local drive.

Regardless, initial import will take time. It's a one time action, so is much less important than say, loading a library view.

It certainly shouldn't be used as the metric to evaluate day-to-day performance in my opinion. It's like gauging operating system performance based on how long the install takes.

JimH · « **Reply #21 on:** September 08, 2009, 12:01:44 pm »

Quote from: globetrotters1 on September 08, 2009, 11:33:18 am

... analyzing activated, thumbnails activated ...

That will affect the speed.

leezer3 · « **Reply #22 on:** September 08, 2009, 12:22:08 pm »

Adding large numbers of database fields at import is always going to take time, especially when they've got a decent amount of content in them.
Minor clarification I suppose, this is a complete *fresh* import, in that the only populated fields are likely to be the basic MP3 data, any thing else or with large amounts of data is going to slow things down like hell!

-Leezer

globetrotters1 · « **Reply #23 on:** September 08, 2009, 03:17:23 pm »

Matt,

not that i want to complain! It's far better to import 5 hours than a whole day in 30'000 track packages... and I have to admit now that your automatic import function is far better and easier than the manual import in packages started via Explorer (which I did before)

And I suppose switching to a quad CPU will make a difference too

I know, my system is not cutting edge, except probably the Areca RAID-6 controller. That's why I want to replace the M/B and CPU as soon as possible. I suppose the upgrade to something like a Asus P5Q Pro/Deluxe or similar and a quad processor will do the job.

When I finally got it installed I'll report the new numbers. Might be interesting to know for yourself too.

INTERACT FORUM

Author Topic: What's likely to be the bottleneck in MC14 with a massive library? (Read 3706 times)