Topic: Manual Import Absurdly Slow (Read 5732 times)

glynor · « **on:** October 13, 2006, 11:17:44 am »

I've noticed that the Manual Import tool is now absurdly slow, especially when the files live on a Network Drive. It seems to be a huge order of magnitude slower than MC11.1's import tool was....

I'm assuming this is because of Otto. In order to prevent performance issues while Otto was running in the background, I assume you somehow limited the CPU usage of the background task (made it a "nicer" task in POSIX parlance).

If these assumptions are correct, is there any way that the "manual import" routine (the one initiated when you go to Tools -- Run Auto Import Now) could be made "less nice" and put back to the way it was in MC11.1, while keeping Otto unobtrusive like it is now?

Otherwise, if my assumptions are wrong... Why is it so darn slow? (If it's just slow because it needs to transfer the files across the network and I'm crazy that MC11.1 seems quicker, just let me know you think I'm crazy and I'll probably test it to see.)

InflatableMouse · « **Reply #1 on:** October 13, 2006, 12:01:42 pm »

MC11: Library now has 4572 files. Search and update took 2:48. (27.21 per second)

MC12: Library now has 4607 files. Search and update took 2:51. (26.94 per second)

This is from a local drive. Can't test network drives as my server is down for maintanance.

Matt · « **Reply #2 on:** October 13, 2006, 12:09:59 pm »

Otto's sneaky, but can't be blamed for this. He can't run during manual import.

What types of files are you importing? Could there be a network change?

As InflatableMouse posted, there shouldn't be much difference. (except that MC 12 is better about working in the background so the UI stays smooth)

Doof · « **Reply #3 on:** October 13, 2006, 12:37:52 pm »

Quote from: Matt on October 13, 2006, 12:09:59 pm

(except that MC 12 is better about working in the background so the UI stays smooth)

I think this is what he means. Is it possible that, because MC12 works better in the background, that the manual Import process runs slower because less CPU time is being devoted to it (to insure the GUI stays smooth)?

JimH · « **Reply #4 on:** October 13, 2006, 12:49:44 pm »

What about thumbnailing during import? Is it set?

Doof · « **Reply #5 on:** October 13, 2006, 01:05:56 pm »

Quote from: JimH on October 13, 2006, 12:49:44 pm

What about thumbnailing during import? Is it set?

It's funny you should mention that. I just ran a manual import (over the network) to my newly built HTPC running MC12 for the first time. I had it set to create thumbnails upon import. The import ran for quite a long time, and when it was done... no thumbnails. I had to have MC manually create thumbnails which then took 2.5 hours to complete.

jgreen · « **Reply #6 on:** October 13, 2006, 01:15:05 pm »

I have also seen "absurdly slow" importation, in a couple of cases. The first was on a USB drive a parent folder containing album subfolders of FLAC and WMA lossless. The second was last night, on a LOCAL drive, same folder and file structure. 73 files took longer to import than to rename from properties to a DIFFERENT IDE drive, once they were imported. Say, 3 per minute.

Not all of the 73 took that long, and that has me scratching away at the old noggin. There was a run of a dozen or so in the middle that was lightning quick, as is usual for MC's great importer function. I looked around a bit at them, and it MIGHT be that the ones that were slow were 48 khz rather than 44.1. Is it possible that MC 12 is slower on file formats that it isn't particularly comfortable with?

glynor · « **Reply #7 on:** October 13, 2006, 01:24:34 pm »

Quote from: JimH on October 13, 2006, 12:49:44 pm

What about thumbnailing during import? Is it set?

No, it is not. However, since they are images, if I switch to the Image view scheme they do build.

This particular import process (which has been going since 10:30 am EDT this morning and it's now 2:22 pm and it has imported 823 of 18,000 files or so) consists entirely of image files of varying types (CR2, JPEG, and TIFF mostly).

I had suspected that the issue was MC12 specific. If Matt says it is not, I'll take him at his word. I do see my network utilization has been hovering at around 60 - 80 % so it could be just that it's just taking that long to transfer them across the network.

JimH · « **Reply #8 on:** October 13, 2006, 01:26:03 pm »

Automatic virus checking?

dcwebman · « **Reply #9 on:** October 13, 2006, 01:38:34 pm »

The speed for MP3 files over a network seems okay to me when I did a manual import in 91.

But I asked in the previous build thread and never saw a response. Can anybody confirm that Otto is working? I had files that I put in the media folders and on starting MC 12 they never got imported. I gave up waiting for Otto after 30 minutes and did the manual import.

glynor · « **Reply #10 on:** October 13, 2006, 01:39:22 pm »

Quote from: JimH on October 13, 2006, 01:26:03 pm

Automatic virus checking?

Not on network drives. Something interesting though...

I just noticed in testing that my import went way, way, way faster if I first disabled Otto. Before, I had it enabled, but manually chose Run Auto-Import Now.

I think the problem might have been that the two import processes were fighting over network resources. Any chance?

I also had the import process crash on me (just stop completely on a file and sit there for 20-30 minutes) three times today.

EDIT: Hmmm.... Actually, I now see that the high network utilization numbers only happen when I'm actively displaying the "images" view scheme (which has a bunch of the un-thumbnailed images visible). When I have anything else displayed (like Audio for example), my Network utilization is very low (near zero with occasional spikes) and the import process is mind-numbingly slow. Here's a screenie:

Click on the image to enlarge.

The server machine is fairly powerful and is otherwise sitting idle. Transferring files off of it works fine and is zippy as ever.

As you can see, for testing purposes I have my Symantec Anti-Virus application disabled.

JimH · « **Reply #11 on:** October 13, 2006, 01:43:46 pm »

Quote from: glynor on October 13, 2006, 01:39:22 pm

I also had the import process crash on me (just stop completely on a file and sit there for 20-30 minutes) three times today.

Could be our problem, but I'm always suspicious of Network, USB, and Firewire drives. We've seen problems in the past that were cured by updating drivers. Maxtor was one.

Matt · « **Reply #12 on:** October 13, 2006, 01:45:30 pm »

Quote from: glynor on October 13, 2006, 01:39:22 pm

Not on network drives. Something interesting though...

I just noticed in testing that my import went way, way, way faster if I first disabled Otto. Before, I had it enabled, but manually chose Run Auto-Import Now.

I think the problem might have been that the two import processes were fighting over network resources. Any chance?

I also had the import process crash on me (just stop completely on a file and sit there for 20-30 minutes) three times today.

Glynor, maybe you should test some files on a local drive.

This sounds like it could be a network problem.

glynor · « **Reply #13 on:** October 13, 2006, 01:48:57 pm »

Ahh... But see, network throughput is fine when I'm thumbnailing in MC. It's just importing that's slow.

And transferring files by copy/pasting through Windows Explorer works fine and gives me good throughput.

JimH · « **Reply #14 on:** October 13, 2006, 01:55:40 pm »

Bugs are funny sometimes. I had a network card once that was fine on copying small files but choked when the file was more than 5MB.

Or a network might support file transfer fine UNTIL it becomes non-stop.

MC can stress a system in ways that other programs may not.

My guess is that some bugs don't appear until the system is run flat out at absolute maximum speed for some period of time. Then they rear their ugly heads.

glynor · « **Reply #15 on:** October 13, 2006, 01:59:16 pm »

I should also mention that I've seen this same behavior on about 4 different machines, with different networks and different drivers. (At home and at work.)

I'll let you know more as I discover it. Something to be aware of and watch out for.

I also just tested it, and MC12 is definitely slower than MC11.

InflatableMouse · « **Reply #16 on:** October 13, 2006, 02:14:44 pm »

In task manager, enable collumns for 'page faults' and 'virtual memory size'. Watch the MC process when its idle, and start the import. See what the numbers do. Page faults will probably start to rise, but in your case I suspect VM size will go up as well.

How much physical memory is in your system?

If needed, we can also troubleshoot with perfmon but that can be a tedious job.

Mr ChriZ · « **Reply #17 on:** October 13, 2006, 02:21:42 pm »

Quote from: InflatableMouse on October 13, 2006, 02:14:44 pm

How much physical memory is in your system?

I guess you've not seen some of the specs on Glynors machines!

I'm sure he should have stocks/shares in Crucial/Kingston etc.

InflatableMouse · « **Reply #18 on:** October 13, 2006, 02:27:12 pm »

Quote from: Mr ChriZ on October 13, 2006, 02:21:42 pm

I guess you've not seen some of the specs on Glynors machines!

In fact I did and now I'm guessing that you don't know how that works?

(just kidding mate)

Commit charge was 930Mb of physical memory, of a total of 2457Mb. That last number includes the pagefile. Its not representative of total physical system memory.

glynor · « **Reply #19 on:** October 13, 2006, 02:51:03 pm »

Quote from: InflatableMouse on October 13, 2006, 02:14:44 pm

How much physical memory is in your system?

This one's actually kinda low, as it's works machine not my personal one. It only has 1 GB of physical RAM (all of mine at home I run at least 2 GB now).

I should also mention, when I took that screenie, I also had some other apps open (but on the second monitor). I was running Adobe Photoshop, Nero 6, and at least a couple of instances of Firefox Bon Echo.

I'll have a new and better screenie in a few minutes...

I might have just figured something out.

InflatableMouse · « **Reply #20 on:** October 13, 2006, 03:10:07 pm »

That's what I suspected and that's why I suggested to watch the vm size collumn.

Although that pc has an issue there, forget about it because after I reread the posts its not the cause for your problem.

glynor · « **Reply #21 on:** October 13, 2006, 03:23:51 pm »

Here's a new screenie:

Click on the image to enlarge it.

What I found interesting was that in watching the import run with Process Explorer open, I noticed a pattern. The import process would go along in fits and starts, much like someone else in this thread described above. It will hum along for 20-30 files or so, and then freeze on one of them for a long period of time. It would then count upwards painstakingly 1 file every 1-2 minutes or so for a while, then suddenly blaze through another group of 20-30 files.

What I noticed, was just as the "fast importing" ended and it got hung up on a certain file, I'd see the DCRaw.exe task show up in Process Explorer. It would only last a second or two and disappear, but the "file count" in MC's import progress dialog wouldn't increment for a very long time, and simultaneously I'd also see the network usage spike for a sustained period. In the screenshot above, the long period of 45-55% network utilization is during one of these periods.

When the "fast importing" periods are happening, the network utilization stays very low (at around 2-3%).

I should mention, it's entirely possible that there is a problem with leaky software on this machine, since it's my work's machine (and there's some software they require that I don't use but that's there anyway -- like a Citrix client). The machine itself works quite well though generally. Here are some stats:

Intel Pentium D 940 (Presler)
Intel D975XBX Motherboard (i975X with ICH7)
1 GB DDR2 at 333 MHz (PC2-5300 Micron RAM, 3:5 ratio, Timings: CAS 5.0, 5, 5, 15, 21)
GPU: nVidia 7800GTX 512MB (ForceWare 91.47)
Windows XP SP2

EDIT: Added some stuff about the machine.

jgreen · « **Reply #22 on:** October 13, 2006, 03:30:05 pm »

Glynor, your "fast/slow" importation is exactly what I noticed last night.

InflatableMouse · « **Reply #23 on:** October 13, 2006, 03:30:46 pm »

When I was typing one of my previous messages, I had a few steps for you to try to troubleshoot. One of em was to exclude RaW files from the import. I choose 'preview post' and saw a red line that 7 new posts had been posted. I erased my post and went back reading the new ones first.

So, there you have it or is there still an issue without those files?

glynor · « **Reply #24 on:** October 13, 2006, 03:35:18 pm »

I don't really have an effective way to test it, as the pictures are all mixed together on the server (the CR2's are next to the JPGs mostly). I will say that I've noticed it occasionally acting retardedly slow on my machine at home, but I don't have any "stats" or "specifics" for that. Generally at home, I don't have many raw images (and any I do were imported into the library a long, long time ago).

bebbetufs · « **Reply #25 on:** January 09, 2007, 03:48:40 pm »

I have this problem with 12.139

six hours to import 1200 mp3s from a network source. and 10.000 to go...As soon as the import process starts the one core of my AMD x2 maxes out. I can still use my pc thanks to the other core and it does not crash...
network utilization varies between 0 and 6% according to task manager. This never happened in MC 11...

glynor · « **Reply #26 on:** January 09, 2007, 03:51:41 pm »

Quote from: bebbetufs on January 09, 2007, 03:48:40 pm

I have this problem with 309

What is 309? The current public build is 12.0.139. Is that what you meant?

MadJewDisaster · « **Reply #27 on:** January 09, 2007, 04:24:21 pm »

Well not sure your case but i had this prob OTTO off - It looks like than drives who are full take ages to import
By full i do not mean == lot of giga to import , but 75 up to 85% of the drive full.
Importing from drives half empty was ok .

dcwebman · « **Reply #28 on:** January 10, 2007, 08:08:15 am »

When doing major editing or importing where the database has to be updated, I have learned to remote desktop into my server that has the media files. Using MC there which then makes it a local drive is SOOOO much faster. I restore my latest backup, make the database changes, back it up, and then restore on my main computer. Until I get a 1GB network, I'll probably need to keep doing it this way.

bebbetufs · « **Reply #29 on:** January 10, 2007, 12:53:00 pm »

Thanks for your replies. (I'm not receiving e-mail notifications for some reason)

Yes I mean 139

I do not see how the paths can be correct if you move a library from a file server on the network to the local machine (htpc)...

I realize that what you describe is much faster, and thats what I've been doing, but I like to keep a library on the (htpc) machine for those few cases where something goes wrong with the network. Although it was slow (as expected) in MC 11 it is utterly useless in MC 12...and besides it does not tax the network at all, it makes the system process go to 50% (one core at 100%) and stay there, but nothing happens on the network.

bebbetufs · « **Reply #30 on:** January 10, 2007, 12:55:59 pm »

By the way, could it have something to do with the cover which I have tagged into the mp3s?

dcwebman · « **Reply #31 on:** January 10, 2007, 06:47:28 pm »

Quote from: bebbetufs on January 10, 2007, 12:53:00 pm

I realize that what you describe is much faster, and thats what I've been doing, but I like to keep a library on the (htpc) machine for those few cases where something goes wrong with the network. Although it was slow (as expected) in MC 11 it is utterly useless in MC 12...and besides it does not tax the network at all, it makes the system process go to 50% (one core at 100%) and stay there, but nothing happens on the network.

I probably wasn't clear but I have a library on both my server and my main computer. I keep it the same by backing up/restoring to the same location on the server. It's only major editing that I go directly to the server to do it. I really haven't seen a difference between the speed over the network between MC 11 and 12.

bebbetufs · « **Reply #32 on:** January 11, 2007, 07:33:55 am »

I see....thanks...

Funny how this is hitting only a few people...I have tweaked my computer a bit, some regtweaks and some services are disabled. Could it be that MC12 requires some services that MC11 does not?

dcwebman · « **Reply #33 on:** January 11, 2007, 08:02:04 am »

You know, I may take back what I said. I had @140 files last night that I put on my server and then decided to import them using my main computer via the network. It was SLOW. I mean we're talking maybe 1-2 files per second as you could sit there and count the number of files. I had to set the computer (laptop) aside and come back to it later.

Now editing the files after they were imported were a lot faster but the import was very slow. This was using 146.

zirum · « **Reply #34 on:** January 11, 2007, 09:44:24 am »

Hmm. I thought i'd just test this on my machine at home just for the fun of it. I have MC11 and MC12 installed on the same machine. Both are currently using local files, but tested to add 100 files from my NAS.

By drag'n'drop (clocked with a wristwatch):
MC12 Exactly 30 seconds
MC11: About 6-7 seconds

This supports what some of you others have been detecting.

Another test: importing a folder with 100 songs, still from NAS, but clocked by MC. I also reversed this test to see if there might be a chaching issue:
MC11: 28 sec
MC12: 29 sec

The NAS is connected on a 100mbit switch. Don't know what to get out of this results thou...

Matt · « **Reply #35 on:** January 11, 2007, 10:28:02 am »

Zirum,

I'm confused about those results. In one test, MC 12 runs at the same speed and in one it's a lot slower.

Can you pinpoint what the difference was? (how the files were added, what types, etc.)

Thanks.

zirum · « **Reply #36 on:** January 11, 2007, 12:21:37 pm »

I agree, the results where weird. I just did another test (similar, I belive), and this time it took about 30 secs for both. I had some problems with the library on MC11 when I tested it, it might have been that it somehow contained both the files from the NAS as well as the files on drive. It kept counting to the double amount (40k) of files I actually had. Even thou it only showed the right amount (20k) in library.

I created a new library, and both imports where equal.

So I withdraw my test results, untill anything else can be proven

bebbetufs · « **Reply #37 on:** January 13, 2007, 01:46:08 pm »

Not that I think it matters but i just upgraded my network to gigabyte speeds and no difference.
I'm currently importing one folder from my server and I swear it only does about one file every 3-4 seconds.

CPU stays around 45-55% one core almost fully loaded.
It is the SYSTEM process which is loading the CPU, media center 12 stays on 0%
Network load is around 0,05 percent.
Server is happy and responsive.

?

JimH · « **Reply #38 on:** January 13, 2007, 04:22:53 pm »

Quote from: bebbetufs on January 13, 2007, 01:46:08 pm

I'm currently importing one folder from my server and I swear it only does about one file every 3-4 seconds.
CPU stays around 45-55% one core almost fully loaded.
It is the SYSTEM process which is loading the CPU, media center 12 stays on 0%
Network load is around 0,05 percent.

Virus checker?

bebbetufs · « **Reply #39 on:** January 13, 2007, 10:14:37 pm »

Yes....NOD32 It is set to not scan MC12. Closing it makes no difference...

Quote

Library now has 6986 files. Search and update took 07:50:24.

Imported 3167 new files.
Failed to import 14 files.

Update. Ran the importer in safe mode:

Quote

Library now has 10982 files. Search and update took 8:48.

Imported 3939 new files.
Failed to import 20 files.

I have found the culprit. It is NOD32 AMON module that is causing the slowdown.
Shutting down NOD32 does not help, that's why I didn't get it earlier. However, by unticking the File system monitor AMON enabled check box in AMON control panel imports run at the same speed as before. Adding Media Center 12.exe to AMON exclusion list does not help.

bebbetufs · « **Reply #40 on:** January 14, 2007, 08:43:28 am »

After a lot of playing about with NOD32 I still haven't been able to find a way to exclude the MC12 import from being scanned by AMON.
Can someone in here point me in the right direction?

Matt · « **Reply #41 on:** January 14, 2007, 03:35:24 pm »

Thanks for letting us know.

Unfortunately, it's common for strange problems to turn out to be from a virus checker or spyware checker.

Sometimes the cure is worse than the disease.

INTERACT FORUM

Author Topic: Manual Import Absurdly Slow (Read 5732 times)