Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[SimonT]

Apologies if I missed the info somewhere, but I seem to be running into some performance issues and I am wondering what a realistic (maximum) library size is?  With music, images and video I have several hundred thousand files (roughly 1.5 TB) spread over various NAS and local drives.  Directories are monitored for changes and I typically get long pauses as I move around the library.  Thumbnails are also slow to load (even after pre-loading them via the tree and view option).

If my library size is realistic, I will look for issues in network communication etc. but I'm interested in how typical my situation is?  Does the architecture of MC allow for this size of data set?

Thanks for your feedback,
Simon T.

[gvanbrunt]

I don't believe that is unreasonalbe. I've heard of others with libraries that large. Also, importing takes place on a background thread, so it should not affect UI performance. Perhaps others with large libraries could chime in. Mine is only a measly 30,000 or so items...

[BryanC]

Mine is about that size with no problems, but I'm using a mix of SATA and USB2.0 drives. Are you running gigabit ethernet?

[SimonT]

Yes I am, but there's no way the NAS drives reach that sort of speed.  On the other hand even a network data rate of a few MBits per second would not explain the sort of delays I see.  If capacity isn't an issue I'm thinking along the lines of the poor Vista performance with network file transfers, could also be the firewall limiting things - I will keep experimenting...  Thanks so far.

[gvanbrunt]

The thing is, I don't believe that any network issues would affect things either as far as the UI is concerned. That is unless you store your Thumbnails there... Don't confuse that with coverart. Thumbnail location can be changed via the registry.

AFAIK, thethe only things that could affect the UI are the db and thumbnails. A dev would have to veryify that. If those are both local, I would think something else on the local computer is slowing it down. Prehaps entering the location of both of those in Virus Scan exceptions would fix it? You could also try shutting down everything other than MC that you don't absolutly need and see if performance improves.

[gvanbrunt]

Oh one more thing. Doubtful, but if you have really short time for shutting down your HD in power settings, it could shut down, then take a bit to spin up and make the UI freeze. I've seen that on mine.

[JimH]

Possibilities...

A virus checker

Library database stored on a network drive -- check MC options for File Locations

Non standard views -- calculated fields can slow things down

[SimonT]

Thanks all - issue is a combination of virus checker and slow RAID drive I think.  I stupidly assumed that the process of generating thumbnails would not involve transferring the complete image across the network, but of course to create the thumbnail, MC has to process the all data first...

[nickeaston]

Upgraded from mc12 to mc14 recently and it appears much slower in UI response and 'importing media'...for no apparent reason.  I have a quarter million tunes in one folder on several external usb raids; my database is on my primary drive.  Nothing in my hardware has changed with my mc upgrade.  Right now it appears that it will take several hours to add 8,000 new tunes to my library...I have always been amazed at the speed of mc12; I do a substantial amount of re-tagging/renaming and was always super-responsive...

[JimH]

Upgraded from mc12 to mc14 recently and it appears much slower in UI response and 'importing media'...for no apparent reason.  I have a quarter million tunes in one folder on several external usb raids; my database is on my primary drive.  Nothing in my hardware has changed with my mc upgrade.  Right now it appears that it will take several hours to add 8,000 new tunes to my library...I have always been amazed at the speed of mc12; I do a substantial amount of re-tagging/renaming and was always super-responsive...

It may be doing work in the background.  It will speed up when it's finished.  You can turn this off if you want.  Check the options for cover art lookup, audio analysis, and thumbnailing (options has a search window in the upper right).

[nickeaston]

All 3 suggested settings are unchecked. I do audio analysis manually after importing.

I am doing an a/b (mc12 vs. mc14, one loaded at a time) comparison now.  MC12 adds new files to the library at about 1 per second; MC14 takes about 8-15+ seconds.

All my media is .mp3; I use no other media in MC.  I still have 7,400 to import; more suggestions are welcome.

At this rate in MC14 import will take about 90,000 seconds (several days).

[Mr ChriZ]

At this rate in MC14 import will take about 90,000 seconds (several days).

This isn't very useful.... but there's 86,400 seconds in a day  
(One of those odd things I just know for no apparent reason....)

For reference MC14 imports on my machine at well over one file a second...

[nickeaston]

Chris, your reply isn't very useful any more than my math...

I provided a comparison between 2 versions of MC which I can run non-concurrently (now running MC12 at faster than 1 second per file import).  As a many-year customer of JRiver, and as a Windows power user, I would like to see some teckkie suggestions please.  Since I am not a programmer, nor can I guess at the differences between MC upgrades, as I said before I am willing to act on suggestions.

For example, my pagefile is set at 6tb max, my system is XPProSP3, cpu 3.19, 3.5gb ram.

Both 12 and 14 libraries are stored on my primary drive.

Corrrection:  Above comment should have been MC12 imports faster than 1 sec/file.

[JimH]

Virus checker?

[Alex B]

nickeaston,

Are you testing this because of pure curiosity or do have some other reason for not restoring an MC12 library backup file on MC14? I would consider an MC library of a quarter of million files valuable. I would not like to lose it.

In fact, I have never lost my main library since Media Center was first released over seven years ago. It has survived through all version upgrades. That's also why I have not recently tested importing a complete media library. I have only added new stuff.

Though, I have noticed that the Auto-Importer seems a bit sluggish when it works as a background process. I think it works faster when it is started manually (you can add the Run Auto-Import Now button to the toolbar and keep the background importer disabled).

[nickeaston]

ZoneAlarm 10 is my resident a/v; when I stopped it and killed the associated processes that I know about, it made no difference.  ZA is of course notorious for leaving junk including running process when shut down.  Did not uninstall it.

Turned it back on and gave MC13 full access in the 'programs' section, identical to MC12; no difference in the slow import rate.

[nickeaston]

Alex:

I DID restore mc12 library to mc14 and all went well; my posts today refer to adding an additional 8k of files to the existing library and the speed of the import process of the 2 versions--the 'adding files to library' process.

I lost my library a couple years ago from a thunderstorm, and am well aware that it takes "days" to start from square one...

I also deactivate the background import feature.

[Alex B]

Oh, I see. I got the impression that you are importing a smaller set of files as a test before doing the "big thing".

I also deactivate the background import feature.

Which import method do you use? Are you just running the auto-import tool by starting it manually?

I could try to reproduce the problem with a few thousand additional files. My base library consists of 85k files, not 250k, but I do have lots of tagged data and that can make the database bigger in "another direction".

[Matt]

What types of files?

One difference is that Media Center 14 imports multiple files at once when run manually.  This should provide (sometimes drastically) better performance on a multi-core system.  If you're using a network that doesn't handle concurrent requests well, it might be choking.

Background imports only import one file at a time and intentionally run slowly to keep background CPU usage low.

[nickeaston]

I always use "import a single folder" and my batches run from a few to several thousand at a time.

My Dell cpu is supposed to be dual core.

I always turn off background import.

All my media files are .mp3.  I rip and convert externally from MC.

I ran 12 and 14 concurrently for a while and side-by-side still showed a 10x difference in importing new files (into their respective database libraries).

The only 'network' involved is an eSATA external raid drive.

[Alex B]

I just did a test. I used the same import method as you. The source drive was an external USB2 drive (WD Elements 1TB) on a quad core AMD Phenom II PC (3.1GHz/4GB/XP SP3).

The result:

Library now has 94404 files. Search and update took 7:23.

Imported 9765 new files.

= 22 files per second. All files were MP3.

Something is definitely not right if your speed is about 8-15 seconds per file.

[nickeaston]

  
Alex:


I downloaded Jukebox12 and am now updating a mc12 library, adding about 30,000 files to a library of about 250,000 at about 1 file per second or less...

Strangely MJ12 wouldn't import any mc14 .zip backups...but opened my mc12 .zip without a problem.
=====
=====
Now running MC14 import: (about 253,000 .mp3 files in 1 directory) on my eSATA

"Searching for files" took about 5 minutes, is now "adding files to library" at about 15 to 20 seconds per file.  I'll reboot the system in the morning and see what happens with mc14.

One hour later, still taking 20-30 sec per file...cpu sucks 30 to 60%...

[JimH]

Check the problems and solutions in the thread called "Weird Problems" in my signature.  You might find something similar.

[Matt]

Thanks to extensive testing by AlexB, we believe the import performance issue is caused by a combination of:
1) A huge number of files in a single folder (over 250,000 in this case)
2) Not using embedded album art

When a file does not have embedded art, Media Center checks on disk for matching art.  In a folder with 250,000+ files, file system operations become very slow.

You might consider using Media Center's "Rename, Move, & Copy Files..." tool to split into artist and album folders.  I would also recommend using internal album art.

We will also see if we can do anything to optimize this case in the program, but 250,000 files in a single folder is not something we've commonly encountered.  We are constrained by the underlying file system performance, which does not like a lot of files in one folder.

[Alex B]

I posted my test results first directly to the development team. Since Matt already referred my tests and because they might be useful for others too I am posting them now here.

Yesterday I posted my MP3 import test results on the MC14 board.

Even though the overall speed was fine, I noticed that there were lots of uneven pauses during the import process. I wanted to test the behavior more before reporting anything. Today I did several additional import tests.


The yesterday's test:
------------------------------------------------------------------------------
Library now has 94404 files. Search and update took 7:23.

Imported 9765 new files.

= about 22.0 files per second
------------------------------------------------------------------------------


The new tests:
------------------------------------------------------------------------------
Library now has 130825 files. Search and update took 31:46.

Imported 36421 new files.

= about 19.1 files per second
------------------------------------------------------------------------------
Library now has 186811 files. Search and update took 43:38.

Imported 55986 new files.

= about 21.4 files per second
------------------------------------------------------------------------------
Library now has 242797 files. Search and update took 48:01.

Imported 55986 new files.

= about 19.4 files per second
------------------------------------------------------------------------------
Library now has 298783 files. Search and update took 49:52.

Imported 55986 new files.

= about 18.7 files per second
------------------------------------------------------------------------------


A smaller test with embedded cover art. I added a 500x500/33kB image
file to 1000 MP3 files. All files were in a single folder:
------------------------------------------------------------------------------
Library now has 299783 files. Search and update took 0:27.

Imported 1000 new files.

= about 37.0 files per second
------------------------------------------------------------------------------
- This result is surprisingly fast. It hesitated a bit before starting,
but after that the file counter didn't pause at all.


The same files as above in a different folder. Now without embedded
art. The image file was in the file folder as "folder.jpg":
------------------------------------------------------------------------------
Library now has 300783 files. Search and update took 0:43.

Imported 1000 new files.

= about 23.3 files per second
------------------------------------------------------------------------------
- Significantly slower than with embedded art


A big set again:
------------------------------------------------------------------------------
Library now has 357769 files. Search and update took 01:08:03.

Imported 56986 new files.

= about 13.95 files per second
------------------------------------------------------------------------------
- It's now slower. The pauses seem to be slightly longer.

- I created a small screenshot video of the final stage of this final test. The video shows how the file counter pauses after every two hundred files or so. I wonder what kind of data my four AMD Phenom II cores running at 3.1 MHz are processing during these pauses and if the speed could be improved. There was quite a bit hard drive activity on the C: drive during these pauses. (The C: drive contains OS/MC/library/temp. The MP3 files were imported from an external USB2 drive.) The video:


http://www.pix01.com/gallery/4AD73041-94E6-4810-80BE-EDB7F3274485/MC_import_pauses/

- an alternative link to the same video (in case this happens to play the video better):
http://s238.photobucket.com/albums/ff132/alexb2k/MC/?action=view&current=mc_import_pauses.flv



I did all tests by importing a single base folder. (I.e. I used the Import a single folder tool). The Auto-import feature was completely disabled. In addition I disabled all user adjustable background tasks (podcasts, servers, logging, last.fm, etc). Also my Avira virus scanner was disabled.


P.S.

I decided to do a few additional tests.


The same 1000 MP3 files as in the earlier two "1000 files" tests,
but now without any cover art:
------------------------------------------------------------------------------
Library now has 358770 files. Search and update took 2:12.

Imported 1000 new files.

= about 7.6 files per second
------------------------------------------------------------------------------


The same 1000 files, except that I added the embedded images back to all files
and imported from a different location.
------------------------------------------------------------------------------
Library now has 359770 files. Search and update took 0:06.

Imported 1000 new files.

= about 166.7 files per second
------------------------------------------------------------------------------
- How this is possible? The files are 1.76 GB. I suppose they can't be in a cache (though, this PC has almost 3 GB of available memory and the images were added outside MC just before the test).


I repeated the above test (with embedded art) three times.
Each time the files were in a new location:
------------------------------------------------------------------------------
Library now has 360770 files. Search and update took 0:10.

Imported 1000 new files.

= about 100 files per second
------------------------------------------------------------------------------
Library now has 361770 files. Search and update took 0:11.

Imported 1000 new files.

= about 91 files per second
------------------------------------------------------------------------------
Library now has 362770 files. Search and update took 0:10.

Imported 1000 new files.

= about 100 files per second
------------------------------------------------------------------------------


Why embedded cover art makes it faster?


P.S. 2

In my opinion MP3 importing is not slow, but perhaps this helps to optimize it also when embedded cover art is not present.

In addition to what Matt said about cover art in the above reply he commented the pause issue as follows:

There is a periodic database save during import so a crash or loss of power won't lose all the work.  This might explain the little pauses that get longer as the library grows.

Since my files contain varying amounts of metadata it is understandable that the pauses may have varying durations.

[nickeaston]

Interesting point about the embedded thumbnails--especially the technique of adding them back...

In my quest to filter out dupes, I have attempted to strip out non-essential tags, including all images and the old v1 tags at the end of the files.  With MC12, my ripped library grew to about 300,000 tunes and I could/can manipulate files by the thousands, retagging and filtering with DoubleKiller, which has far greater capability than any other duping app with its numerous options.  By tweaking its options, I have in the last 2 years reduced my library to about 250,000, with MC12 and my old dual-core CPUs, and with still, by my guess, about 30,000 sound-alike dupes which I don't want, which no known app can filter out.

An apparently defunct startup app called "Sloud Music Content Inspector" held the greatest promise for me but their beta app crashes both my XP desktops on test runs.  Searching on "sloud" will produce their promising website and their prospectus, which details their proposed technical approach to de-duping:

"...the core system is written in portable C++ programming language and does not depend on any particular operating system.  The client interface is written in C++ for Microsoft(R) Windows(TM) family of operating systems. Real-time recognition or voice pitch, based on algorithms without use of Fast Fourier Transform (FFT). The technology can work on systems with severely limited resources, such as smart mobile phones.  Algorithms of pitch correction of recognized sounds with respect to psycho-physiological characteristics of human audio sensory system. The algorithms improve the quality of sound recognition.  Calculation of sound duration and composition of MIDI score based on the recognized sound. Algorithms of automatic correction of MIDI score in order to remove false scores and improve overall recognition of the music."

I have noticed over the years that de-duping is not a big concern here on the Forums, especially in libraries the size of mine.  I don't claim to know much about the mp3 technical format but I notice that in the MC database (calculated) columns there is frequently a substantial non-correlation between "duration" and "file size" which doesn't make sense to me...  For example, if the "duration" is in fact a calculation of only audio content less beginning and ending blank space, this calculation would go a long way in de-duping...  Of course it still doesn't account for discrepancies in the 'content' from encoding/converting various formats into mp3. Thus my attempts to "clean" my tags are an incomplete and vain attempt to delete "sound-alike" duplicates.  Said in another way, it is my perception that narrowing down "duration" to actual audio content only could help in the final screening for dupes--is this possible?  (re-reading this tells me that silent space would have to be removed somehow to arrive at a true duration)

Today's strategy is to top off my MC12 library (where virtually no files have thumbnails), then import the library into MC14 (shouldn't have to 'add new files'), then add my next large batch of rips, which won't have images removed, and check the 'add' rate...does this make sense?

[Matt]

We recommend splitting your files into folders.  Media Center has the tools to make this easy.

File systems don't like a lot of files in one folder (250,000+ in this case).

We're developing a caching layer for v15 to help import performance in your very specific case, but this won't be available for several months.

[Alex B]

nickeaston,

As a workaround, if you don't care about cover art, you could add a very small generic image file to each new set of MP3 files before importing and enjoy perhaps 100x faster import speed.

Here's an example, the familiar blue note icon: . It is 341 bytes and it might fit in the so called padding area that is often reserved for tag changes in MP3 files. If a tag change fits in the padding area a tag update is very fast. You can use some freeware external tagger for adding the images. For instance, Mp3tag can easily tag a few thousand files at at time.

However, as Matt said, I think it would be a good idea to split the files into folders. You could use something like the release year or alphabet (A, B, C, etc) in the folder names if you don't want to use the usual artist/album structure.

[nickeaston]

My secret solution to handling large numbers of files, especially in one folder, is PDFind, found within the last few revs of Avanquest/PowerDesk.  I break out the app "pdfind.exe" and keep it available on my desktop; neither Win Explorer nor any other file-handling app I have ever tried measure up to handling 2 to 3 hundred thousand files in one folder.  PDFind does it fast and flawlessly. Shareware versions also should have the applet within...

[nickeaston]

Rick, you may be straying off topic (capacity vs performance).  Here are my observations to date:

re: importing new files into existing library--mc12 and mj12 both imported my latest batch of 1800 files, all with embedded images, at the very satisfactory rate of about 1 per second.  mc14 still takes 10 to 15 seconds, for whatever reason, I don't care, because mc12 if necessary will serve me well indefinitely (I concede that mc is the ONLY game in town for managing large collections--if you want to use multiple folders, that is YOUR workaround--I just prefer to use the third party utility I mentioned, but it's not necessary unless one wants to open/browse a huge folder for some reason...  (to be clear, once imported, mc14 works fine for me but adds no significant file management features for a narrow-need user like myself)

[nickeaston]

I might be missing something important, but I see no benefit (to me) of having upteen music folders...file management capability is and has been superb with/within MC for years.  My wishlist is very small--better duping and silence removal would be helpful.

[nickeaston]

Jim, I will do the separate directores thing with my mc14 install, to help complete the thread--how many folders do you suggest for about 280,000 files?

[JimH]

I don't know.  You could try two, then four, then eight.  See if you can find a point at which it makes a difference.   Please post anything you find.

[nickeaston]

Is there any 'overkill' point?  Shouldn't be too hard to create dozens or hundreds in not too much time...

Is this primarily a long-existing Windows deficiency?  Or the new MC?  Or all of the above?

[rick.ca]

Is this primarily a long-existing Windows deficiency?

It's much worse than that. They actually designed the file system to use folders.

[nickeaston]

With MC14 and my files divided up in about 6 directories, "adding" time was even higher (20 to 30 sec/each) than with just one directory (and my 5 year old hdwe)

I currently have two 5 year old machines, my XP machine and a new install of W7 Ult 32bit, both running MC15 Beta, and the average "adding files" rate is well under 1 file/sec. for each machine. (Same music library (1 directory quarter million files), both USB external drives)

I don't think I have any further issues which haven't been resolved with MC15...