INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: very slow and memory intensive import on large library  (Read 983 times)

barchar

  • Recent member
  • *
  • Posts: 10
very slow and memory intensive import on large library
« on: April 03, 2020, 07:19:34 pm »
I have a large library with a bunch of special snowflake filenames (very long names, wierd characters, CJS names, etc) and currently it has 85726 files in it, including 23421 videos and 62293 images. The files are stored on a network share (fedora 31, mdadm raid5, shared with smb). Windows defender was also using up a ton of bandwidth but I think I got the exclusions and it's stopped. MC is still being slow though.

Now when I go to finish the import (there are some more files there) it imports 11 files and then starts to take a LOOOOONG time per file, and use a vast quantity of both memory and network bandwidth (on the order of all my memory). eventually I just kill it because I'm out of memory and it's not making much progress.

Any ideas what's happening? For reference there are about 34,000 total video files in the folder that should be imported (including the 23,000 that have already been successfully imported)
Logged

tij

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1557
Re: very slow and memory intensive import on large library
« Reply #1 on: April 03, 2020, 10:51:51 pm »
Check your analyse audio settings ... it might be turn on for import

Basically MC analysis audio to determine peak values and dynamic range ... so later you can do volume levelling
Logged
HTPC: Win11 Pro, MC: latest 31(64b), NV Driver: v425.31, CPU: i9-12900K, 32GB RAM, GeForce: 2080ti
Screen: LG 2016 E6
NAS: FreeNAS 11.1, SuperMicro SSG-5048R-E1CR36L, E5-1620v4, 64GB ECC RAM, 18xUltrastar He12-SAS3 drives, 2x240GB SSD (OS)

marko

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8955
Re: very slow and memory intensive import on large library
« Reply #2 on: April 03, 2020, 11:47:04 pm »
Also, check your settings under "Help > Logging"

If that's turned on with no size limit, the file can get really big, which slows the import/thumbnailing process down considerably.
Logged

barchar

  • Recent member
  • *
  • Posts: 10
Re: very slow and memory intensive import on large library
« Reply #3 on: April 04, 2020, 03:16:31 am »
No Luck with either.
Logged

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1852
Re: very slow and memory intensive import on large library
« Reply #4 on: April 04, 2020, 04:51:16 am »
I see the same.  Have been reluctant to raise it as I'm guessing it's may be my NAS, or at least the way MC interacts with it (no other software shows similar issues).   But it is hugely frustrating.

I'm also seeing massive delays when I paste cover art from clipboards into files (trying to diagnose that now).

Looking at the logs it seems MC does a lot of refreshing of the currently display as each file is imported - I've managed to speed things up a little be ensuring I import using an empty view with no smartlist rules (most of my views are complex and continually reevaluating them is slow) but MC still locks for 20-30 seconds at a time.

0049937: 21792: Database: CTagSaveHelper::Thread: Done saving tag
0051937: 21792: Database: CTagSaveHelper::Thread: Finish (15375 ms)
0151421: 21944: Playback: CMJWaveFeeder::Thread: Finished feeder loop (bCancel: 0, bPlayed: 1)


You can see a 15.3s "lock" here, but the logs don't really help [me]...  I'm going to start a separate thread....
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: very slow and memory intensive import on large library
« Reply #5 on: April 04, 2020, 03:23:15 pm »
For movies and TV, MC is going to be looking up the MovieDB and TheTVDB respectively sites as well. As the internet is being hammered at the moment that could be seeing long delays.

Basically turn off all additional work the import process needs to do, and then run those processes later.

Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1852
Re: very slow and memory intensive import on large library
« Reply #6 on: April 05, 2020, 01:51:31 am »
As noted, I see the same issue with importing.  I created an empty library, as per JimH's suggestion in my thread, and the problem was gone - everything was instant, so to me it seems to be library related.

One thing I did notice was that:

C:\Users\<user>\AppData\Roaming\J River\Media Center 26\Temp

Was full of hundreds, if not thousands, of temp cover art files.  I deleted them all and this seems to have helped a little.  The OP might check this folder on their machine.

I've sent my library to Matt and asked him to take a peek.  If he can find, and fix, any issues, it might resolve the problems here as well...
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: very slow and memory intensive import on large library
« Reply #7 on: April 05, 2020, 02:14:24 am »
Ah, one of those issues. I'm going to post over in your thread because yours is no doubt an old Library updated many times with MC, rather than just a big Library.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

S. Pupp

  • Junior Woodchuck
  • **
  • Posts: 83
Re: very slow and memory intensive import on large library
« Reply #8 on: April 05, 2020, 09:32:35 am »
I had the same issue.  I then reinstalled Windows 10 and tried again with an empty library, but with the same issue.  Windows Task Manager showed that Media Center was using about 26GB of my 32GB RAM.  Shortly after, Windows gave low memory messages.  It took several reboots to complete the import.  All files were on local drives.  No problems since then.
Logged

barchar

  • Recent member
  • *
  • Posts: 10
Re: very slow and memory intensive import on large library
« Reply #9 on: April 05, 2020, 03:28:30 pm »
I'll try deleting those temp libraries. This happens the very first time I import to a new library, although it gets through several tens of thousands of files before slowing down.
Logged

BigSpider

  • Galactic Citizen
  • ****
  • Posts: 350
Re: very slow and memory intensive import on large library
« Reply #10 on: April 05, 2020, 10:33:34 pm »
This new waveform might be the problem. Switch off audio analysis during import to see.
Logged
English spiders are kind of small, but when I was living in the African rain forest well ......

BigSpider

  • Galactic Citizen
  • ****
  • Posts: 350
Re: very slow and memory intensive import on large library
« Reply #11 on: April 05, 2020, 10:59:05 pm »
Just done a test and definitivly it is the waveform analysis which is causing the problem. And I'd like without what if scenario".
Logged
English spiders are kind of small, but when I was living in the African rain forest well ......

barchar

  • Recent member
  • *
  • Posts: 10
Re: very slow and memory intensive import on large library
« Reply #12 on: April 05, 2020, 11:16:09 pm »
I don't have any of the processing options on.....
Logged

barchar

  • Recent member
  • *
  • Posts: 10
Re: very slow and memory intensive import on large library
« Reply #13 on: April 24, 2020, 03:20:23 pm »
Update! I think I fixed it. I switched my network share from using CIFS (via samba) to use nfs (and mounting it on windows with the builtin nfs support) and things seem to work better. I suspect it has _something_ to do with long paths on my share.
Logged

barchar

  • Recent member
  • *
  • Posts: 10
Re: very slow and memory intensive import on large library
« Reply #14 on: April 24, 2020, 08:52:44 pm »
AAAAND that didn't actually fix it. Next I'm trying again after I've renamed all files on that volume to no longer contain cursed characters (<>|*"?:)
Logged
Pages: [1]   Go Up
« previous next »