Topic: Slow 'Adding files to library' (Read 4444 times)

David_1cog · « **on:** November 24, 2005, 01:53:02 pm »

11.1.65.

My library became corrupted earlier today - not sure how. So, I rebuilt library. The initial scan went OK, but now it's doing the 'Adding files to library' step, and each MP3 is taking just under a second to be added. With ~15,000 files, it's going to be a looooong process.

Is this (lack!) of performance normal?

P.S. Recent builds have been *much* quicker, especially 'Library Browser' scrolling.

David_1cog · « **Reply #1 on:** January 31, 2006, 06:17:24 am »

Bump. This is still an issue with 11.1.108, ~1 second to add each file. It's happening with any MP3 library import. So, next time I need to rebuild my entire library (~20,000 MP3s), it's going to take over 5 hours!

Alex B · « **Reply #2 on:** January 31, 2006, 07:08:59 am »

A bit similar issue was discussed here:

http://yabb.jriver.com/interact/index.php?topic=29251.0

Do you find any similarities with your setup & MP3 file formats?

David_1cog · « **Reply #3 on:** January 31, 2006, 08:00:36 am »

I can't see anything that applies from that thread: I'm not on a network drive, I'm using two local IDE drives. I'm not auto-importing. There was mention of switching off "audio analysis on import", but no explanation on how to do that, and I can't find that setting. I'm seeing similar slow performance for CBR and VBR MP3s.

KingSparta · « **Reply #4 on:** January 31, 2006, 10:37:43 am »

If you think MC is slow Try MM And Real, Now thats slow.

Normaly They Just Crap Out On Me During Tests.

I did notice a Bug I Think the other day in mc where it took like 2 hours, but Media Center Reported It Took 7 Hours (Not Sure What Version But One Of The Latest Builds).

David_1cog · « **Reply #5 on:** January 31, 2006, 10:53:24 am »

I'm sure MC is better than MM and Real in *every* respect, but I'm comparing (from all-too-fallible memory) MC11.0 to MC11.1, and I'm pretty sure this slow down during the 'Adding files to library' stage has occurred recently (last few months).

A 3GHz / 1GB rig shouldn't take 1 second to add a record to a database, which is effectively what this process is doing.

Matt · « **Reply #6 on:** January 31, 2006, 11:00:55 am »

I can't reproduce the slowness.

Impoorting over the network with a debug build and full logging on is still pretty snappy.

Is it possible you have a drive speed issue? (cabling, UDMA, etc.)

Also, do you have VBR MP3 files without VBR headers? These can be a little slower to analyze.

BTW, MC now makes automatic backups so if anything ever happens to your library, you can load from one of those.

hit_ny · « **Reply #7 on:** January 31, 2006, 11:12:50 am »

Quote from: Matt on January 31, 2006, 11:00:55 am

BTW, MC now makes automatic backups so if anything ever happens to your library, you can load from one of those.

I'm still on 11.0 and see MC saving (often).

Is that what you mean by making automatic backups ?

David_1cog · « **Reply #8 on:** January 31, 2006, 11:42:29 am »

I see no other slow behaviour on the drives in question, and I use both heavily for various apps / backups (note: I did make sure nothing else was accessing the drives while importing).

Not sure how to check if my VBR MP3s have VBR headers, but most of my VBRs are ripped using MC on my machine - so assume that MC does the right thing with respect headers.

The auto library backups are good, but still leaves me with syrup slow imports. I'll try disabling antivirus (NOD32) next time, see if that has any impact.

LonWar · « **Reply #9 on:** January 31, 2006, 12:02:07 pm »

Quote from: hit_ny on January 31, 2006, 11:12:50 am

I'm still on 11.0 and see MC saving (often).

Is that what you mean by making automatic backups ?

Auto backups were added in 11.1

11.1.67 (11/30/05)
9. NEW: Media Center automatically creates library backups every 3 days, and also creates monthly images. (and intelligently cleans up old backups)

11.1 Changelog is here: http://yabb.jriver.com/interact/index.php?topic=31545.0

David_1cog · « **Reply #10 on:** February 19, 2006, 06:19:53 am »

11.1.122

My problems with this continues.

I just cleaned file names for 300 MP3s by using a rename utility external to MC. I then selected MC Library ... Import Media (with all file types selected and only 'Fix or remove broken links' and 'Update library for external changes' selected):

'Adding files to library ...' processed the first few tracks ~1 second but quickly slowed down to one track every ~5 seconds. That performance was repeated in the 'Updating library to match ...' stage.

CPU usage showed MC flicking between 0% and 2%, with the System process at ~97%. I disabled antivirus, Copernic Desktop and anything else that might access CPU or disk. A soon as the import finished, System process dropped back to 0%.

Here's some output from the debug log:

Code: [Select]

1225594: Import: ImportNewFiles: Adding: D:\Music\Billie Holiday\1945-1959 - The Complete Billie Holiday On Verve vol.9\Billie Holiday - Studio Talk.mp3
1225735: Import: JRAnalyzer::SetParseIntoPlaylists: Start
1225875: Import: JRAnalyzer::SetParseIntoPlaylists: Finish (140 ms)
1226016: Import: JRAnalyzer::SetAcceptDataFiles: Start
1226172: Import: JRAnalyzer::SetAcceptDataFiles: Finish (156 ms)
1226297: Import: JRAnalyzer::SetCDAInfoMode: Start
1226422: Import: JRAnalyzer::SetCDAInfoMode: Finish (125 ms)
1226547: Import: JRAnalyzer::SetDisableTagReading: Start
1226688: Import: JRAnalyzer::SetDisableTagReading: Finish (141 ms)
1226829: Import: JRAnalyzer::Open: Start
1226969: Import: JRAnalyzer::RemoveAllFiles: Start
1227094: Import: JRAnalyzer::RemoveAllFiles: Finish (125 ms)
1227235: Import: JRAnalyzer::AddFile: Start
1227375: Import: JRAnalyzer::AddFileMJ: Start
1227485: Import: JRAnalyzer::CreateFile: Start
1227610: Import: JRAnalyzer::CreateFile: Finish (125 ms)
1227750: Import: JRAnalyzer::AddFileMJ: Finish (375 ms)
1227907: Import: JRAnalyzer::AddFile: Finish (672 ms)
1228047: Import: JRAnalyzer::Open: Finish (1218 ms)
1228204: Import: JRAnalyzer::GetFile: Start
1228329: Import: JRAnalyzer::GetFile: Finish (125 ms)
1228454: Import: JRAnalyzer::GetPlugin: Start
1228563: Import: JRAnalyzer::GetCanTagFile: Start
1228704: Import: JRAnalyzer::GetCanTagFile: Finish (141 ms)
1228860: Import: JRAnalyzer::GetPlugin: Finish (406 ms)
1229032: Import: JRAnalyzer::GetFile: Start
1229172: Import: JRAnalyzer::GetFile: Finish (140 ms)
1229297: Import: JRAnalyzer::Open: Start
1229422: Import: JRAnalyzer::RemoveAllFiles: Start
1229547: Import: JRAnalyzer::RemoveAllFiles: Finish (125 ms)
1229672: Import: JRAnalyzer::AddFile: Start
1229829: Import: JRAnalyzer::AddFileData: Start
1229969: Import: JRAnalyzer::CreateFile: Start
1230110: Import: JRAnalyzer::CreateFile: Finish (141 ms)
1230250: Import: JRAnalyzer::AddFileData: Finish (421 ms)
1230375: Import: JRAnalyzer::AddFile: Finish (703 ms)
1230500: Import: JRAnalyzer::Open: Finish (1203 ms)

JimH · « **Reply #11 on:** February 19, 2006, 06:44:29 am »

Check this thread for anything related to anitvirus, firewall, etc. Disabling may not be enough.

http://yabb.jriver.com/interact/index.php?board=3;action=display;threadid=24031;start=0

Is the drive a network or USB drive?

Matt · « **Reply #12 on:** February 19, 2006, 09:05:06 am »

Thanks for the log.

However, I wish it made more sense to me. For example, this code is doing almost nothing:

1225735: Import: JRAnalyzer::SetParseIntoPlaylists: Start
1225875: Import: JRAnalyzer::SetParseIntoPlaylists: Finish (140 ms)
1226016: Import: JRAnalyzer::SetAcceptDataFiles: Start
1226172: Import: JRAnalyzer::SetAcceptDataFiles: Finish (156 ms)
1226297: Import: JRAnalyzer::SetCDAInfoMode: Start
1226422: Import: JRAnalyzer::SetCDAInfoMode: Finish (125 ms)
1226547: Import: JRAnalyzer::SetDisableTagReading: Start
1226688: Import: JRAnalyzer::SetDisableTagReading: Finish (141 ms)

Yet it takes over half a second. It's like there's something slowing down COM calls on your system.

Here's the same chunk of log from my system:

0015157: Import: JRAnalyzer::SetParseIntoPlaylists: Start
0015157: Import: JRAnalyzer::SetParseIntoPlaylists: Finish (0 ms)
0015157: Import: JRAnalyzer::SetAcceptDataFiles: Start
0015157: Import: JRAnalyzer::SetAcceptDataFiles: Finish (0 ms)
0015157: Import: JRAnalyzer::SetCDAInfoMode: Start
0015157: Import: JRAnalyzer::SetCDAInfoMode: Finish (0 ms)
0015157: Import: JRAnalyzer::SetDisableTagReading: Start
0015157: Import: JRAnalyzer::SetDisableTagReading: Finish (0 ms)

Any other clues what could slow down basic code? It makes me wonder if there's a virus or anti-virus being overly aggressive.

We'll take a close look on our end too.

David_1cog · « **Reply #13 on:** February 19, 2006, 09:32:39 am »

Jim

The drive is not on a network. It's a slave IDE in a standalone PC. Note, I've also tried the master drive in the PC with same results.

I couldn't find anything in the 'Weird and wonderful' list applicable to my problem / system.

I've just run a variety of benchamrks and analyses against the drive / CPU and it all appears OK. All other apps run OK on the system, including ripping, burning, audio + DVD, image editing, disk indexing, defragging, antivirus scans, etc. etc.

Is there any other information I can provide to help track this down?

David_1cog · « **Reply #14 on:** February 19, 2006, 09:51:10 am »

Matt

I'm open to suggestions, but I unloaded NOD32 antivirus to produce that log, and the resident kernel showed no activity while performing the import (FWIW). I run regular virus scans and my rig appears to be clean.

I've no idea what else could cause this. Let me know if you want any more of my system information.

glynor · « **Reply #15 on:** February 19, 2006, 10:39:57 am »

I'll assume you've probably thought of this, but just in case...

Have you done a thorough spyware sweep and a check for rootkits? I just ask because a rootkit could conceal either spyware or virus activity, and the kernel-mode drivers can cause general system slowdowns (particularly with disk access). Badly-coded spyware is also somewhat notorious for that.

As I said, you probably have, but if not, I'd probably run a check with Sysinternal's Rootkit Revealer. Then run through a pass of Spybot S&D, reboot, and then AdAware.

David_1cog · « **Reply #16 on:** February 19, 2006, 11:52:57 am »

Yep, I run Spybot and Adaware as well as NOD32 and ZoneAlarm Pro.

I tried the Rootkit Revealer - all clear.

Thanks for the suggestions.

Alex B · « **Reply #17 on:** February 19, 2006, 01:22:15 pm »

I have no idea what causes your problem, but until the solution is found you could try these suggestions:

You may want to untick all advanced "smart" options in the import dialog. MC does a lot of possibly unnecessary comparing if they are enabled. Alternatively, after the initial mass import you could use right-click import in Windows Explorer (enable the shell integration in MC options first) or in Drives&Devices (inside MC) for adding new files.

If you changed the tags outside MC do "Update Library (from tags)" instead of importing.

If you changed the filenames too it may be easiest to import the files as new library items. Include only the needed folders. Remove the old files from the library first. This search finds the missing files: [=IsMissing([Filename])]=1

I can understand that sometimes external taggers may be needed for making the files compatible with some other programs, but MC has about the best possible tools for renaming files so it may be better to let MC do that part.

jgreen · « **Reply #18 on:** February 19, 2006, 01:34:31 pm »

My money is on ZoneAlarm Pro, FWIW. The Pro version includes "tools" to monitor/inhibit communication between apps. Try going into the program control module and right-clicking on MC. Go to "options" and make sure "Allow Application Interaction" is Enabled. Also, it would not hurt to give MC "Super Trusted" status (3 bars).

David_1cog · « **Reply #19 on:** February 20, 2006, 09:31:52 am »

* the only 'smart' options enabled are 'fix or remove broken links' and 'update library for external changes', both of which are essential

* Windows right click + import gives same performance, and I prefer to use 'global' Import Media function to pick up changes and new files, manually added cover art, etc.

* I'm not changing tags outside MC, just occasionally renaming files (using Flexible Renamer, which allows me to easily clean / reformat file names)

* ZoneAlarm Pro - I switched off the 'OSFirewall' which caused every application to take a *long* time to open. I've not noticed a performance hit with any other apps. However, I have tried the MC Import with ZA shut down - same results.

... an hour later ...

OK, problem solved. It was NOD32. Even though I shut down the 'Control Center', I didn't realise file monitoring was still active. Once I switched off the 'AMON' file system monitor, MC import was lightning quick again. To fix the problem just for MC (i.e. not disabling antivirus completely), I did the following:

1. NOD32 Control Center
2. AMON ... Setup
3. Exclusions
4. Add ... Folder ... C:\DOCUMENTS AND SETTINGS\USERNAME\APPLICATION DATA\J RIVER\MEDIA CENTER 11\

Thanks to everyone for their suggestions.

Matt · « **Reply #20 on:** February 20, 2006, 12:10:34 pm »

I think the cure is often worse than the disease with virus checkers. Making basic COM calls take a few extra 100ms is completely unacceptable.

Anyway, glad to hear you have it fixed.

David_1cog · « **Reply #21 on:** February 20, 2006, 01:14:19 pm »

Agreed, Matt - I've flagged it with the NOD32 developers. Maybe they'll look in to it.

In NOD32's defence, this is the only performance issue I've had with it ... and it's the only antivirus on the market with 100% VBulletin record.

KingSparta · « **Reply #22 on:** February 20, 2006, 01:53:48 pm »

Quote

and it's the only antivirus on the market with 100% VBulletin record.

I had to disable everything from Symantec It just slows everything down to include network connections to you system.

There is too much security when it keeps programs or your system from running properly.

JaysonG · « **Reply #23 on:** April 08, 2006, 12:32:05 am »

Thanks for this.

INTERACT FORUM

Author Topic: Slow 'Adding files to library' (Read 4444 times)

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog

David_1cog