INTERACT FORUM

More => Old Versions => JRiver Media Center 27 for Windows => Topic started by: rolf_eigenheer on November 22, 2020, 09:18:38 am

Title: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 22, 2020, 09:18:38 am

During the investigation into why the import sometimes takes a long time or is interrupted for a long time, it was noticed that MC is simultaneously accessing other shares of the NAS.

On closer examination, I now find that thousands of files that are not included in the DB are nevertheless queried with QueryBasicInformationFile.

The folder in question may have been opened or read into MC once. Now they are not in the DB but still queried.

Note:
No filters are applied to 'Disk Locatio' View shown below.
No drive mappings are made to \\as6\alex
'Update Tags when File Info Changes' is unchecked

 

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 23, 2020, 01:59:32 am

If the files have been read into MC at some time, and then removed, they may still be listed in the MC "Remove Database". In which case, MC may still be checking if they exist, and if not, removing them from the "Remove Database".

Check if they are in that database, and if so, remove them and see if MC still looks at them.

I haven't checked if this is likely, but it is a possibility.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 23, 2020, 02:14:37 am

This would make sense.
What is the easiest way to clear the 'Remove Database' ?

Title: Re: MC scans files which are not in the DB
Post by: wer on November 23, 2020, 02:37:49 am

https://yabb.jriver.com/interact/index.php?topic=60732.0

Use the smartlist to identify them, and then delete them. Sometimes you have to delete them twice.

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 23, 2020, 02:43:33 am

Just create a new blank View with the rule;
Limit database to: Removed

Which in the Import/Export sub-dialogue of the "set rules for file display" dialogue translates to;
~d=r

Make sure that "Use parent scheme rules for file display" is unticked in the "set rules for file display" dialogue, or create the View in a new root View group that doesn't have any filters in it. I have a "Tools" top level View group for such Views.

Then select all the files that you want to remove in the View, hit delete, and select "Remove from Media Library". Refresh the View and check that they were removed.

The test for the issue again.



EDIT: Or you could use a Smartlist as Wer says.  ;)

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 23, 2020, 03:27:42 am

Ok got it.
It seems to be much easier to get files into the database than to get rid of them ;-) So it seems to be important to be careful already during the import that nothing superfluous is included. This is still clear in the import dialog. But how does it work when importing a directory with the context menu of the Explorer? How can this be configured?

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 23, 2020, 04:10:50 am

When you use MC Explorer to view files in a directory, it seems like MC uses the settings in Auto Import. So if you then imported a file (or directory I guess) using the right-click context menu or button in MC Explorer, I guess it would use the same settings.

But I haven't tested that to prove it is the case. Though I did do one test with Thumbnailing and Audio Analysis and most other tasks turned off in Auto Import, and the long showed Audio Analysis had been done, or Thumbnails created, so that is a good indication. I just wasn't very thorough in the test.

If it is the case, configure through the Auto Import settings. If not, probably not configurable.

BTW I don't think those files you listed are actually being imported by MC. They are just being checked for some reason. So nothing superfluous should be imported, but they are included in the processing, so there is an impact on performance.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 23, 2020, 10:28:35 am

I found 190'000 'removed' files in my DB.
Deleted them all and checked that no more are listed.

Immediatly after that, MC started scanning ALL my Drives (30 TB in total). I can see no reason for that. No auto import is set. 

After restart MC again scans files which are NOT in DB. See attachment.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 23, 2020, 11:40:16 am

MC25 loads files when a view is changed or a playlist is playing. But it does not that background operations.
MC25 feels more responsive.

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 23, 2020, 03:59:47 pm

I was going to say that I have no more ideas, and over to JRiver...

But then I looked back at the first image in this thread, and noted that you were displaying a View which used the "File Path" Category type in its definition. I don't know what the File Path Category actually does, under the covers, but I wonder if it is scanning all drives and network locations to offer them in the View, or something like that. I always thought it just used the locations in the Library, because it already knows about those. But maybe it is doing more.

If you restart MC and don't open any View that uses a File Path Category, does the background scanning still happen?

Title: Re: MC scans files which are not in the DB
Post by: glynor on November 23, 2020, 04:13:03 pm

No. The Filesystem Location Category is just doing a List Type view on the [Filename (Path)] field with the filesystem slash used as the delimiter. You can make it yourself manually using an Expression if you want.

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 23, 2020, 05:15:57 pm

Well, that is what I thought, but the built-in functions such as "Location" are opaque, so I couldn't be sure.

Any ideas about what is actually going on Glynor?

Title: Re: MC scans files which are not in the DB
Post by: glynor on November 23, 2020, 06:00:17 pm

No. That's weird.

rolf, did you check All Databases with a View as shown in my thread here:
https://yabb.jriver.com/interact/index.php/topic,127633.msg885433.html#msg885433

As mentioned there, you have to remove files 2x sometimes (because of stacked files, and for vague other reasons). What about if you load a fresh, cleared Library?

PS. If it is iterating over the filesystem in a buggy sort of way, I bet it is searching for some of the recently (relatively) expanded "Related Files" stuff (or whatever that stuff is called), and the search has gone haywire somehow.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 12:58:59 am

I created the view shown in the attachment. It showed 190'000 files. I removed them and started MC again. Now the view only shows one entry. This is correct because I deleted it later just for testing. So We can be pretty sure, that no references to \\alex should be in the db.

MC25 reads the files which are to be shown in the opened view. After 175 File operations on \\AS6 there are no more activities.
MC27 starts with the 'Start' View. No other view is open. Then I notice that some of my MOVIE drives spin up. The Log shows access to \\as6\audio, \\as6\movies1 and \\as6\movies2 so far. 18'000 file system access to \\as6 within a few minutes. And going on ...

Am I the only one where this happens ? Has anyone else run the logger from sysinternals ?

Title: Re: MC scans files which are not in the DB
Post by: zybex on November 24, 2020, 02:21:25 am

What about MC's own debug logs? It should have clues about what it's doing.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 02:58:37 am

Quote from: zybex on November 24, 2020, 02:21:25 am

What about MC's own debug logs? It should have clues about what it's doing.

Since I have no idea what information is collected by MC, I don't want to share logs and libs in public. Is there a way to send it directly to you ?

Title: Re: MC scans files which are not in the DB
Post by: zybex on November 24, 2020, 03:06:27 am

You can upload it to some sharing service like filebin.net and give me the URL via PM.
You can also open the logs in Notepad++ and try to figure it out :)

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 03:59:52 am

Quote from: zybex on November 24, 2020, 03:06:27 am

You can upload it to some sharing service like filebin.net and give me the URL via PM.
You can also open the logs in Notepad++ and try to figure it out :)

Sent a PM with wetransfer link . Did you receive it ?

Title: Re: MC scans files which are not in the DB
Post by: zybex on November 24, 2020, 04:34:45 am

It's this background thread: GetMissingAndUpdateFiles, which is a sub-task of the Import thread:

0007807: 14300: Import: Process: Starting import (Media Center 27.0.34 (64-bit))
0007808: 14300: Import: GetNewFiles: Start
0007898: 14300: Import: GetNewFiles: Finish (90 ms)
0007898: 14300: Import: GetMissingAndUpdateFiles: Start
0007898: 14300: Import: GetMissingAndUpdateFiles: Getting files
0007898: 14300: Import: GetMissingAndUpdateFiles: Sorting files
0007898: 14300: Database: CMediaFileIOLoad::Load: Loading: C:\Users\Rolf\AppData\Roaming\J River\Media Center 27\Library\Cache\File List Sort\Index.dat
0007900: 14300: Database: CDataHolder::Load: Field: Filename; Files: 4846; Pointer bytes: 39056; Data bytes: 32904; Elapsed ms: 1.308
0007902: 14300: Database: CDataHolder::Load: Field: Name; Files: 4846; Pointer bytes: 39056; Data bytes: 82056; Elapsed ms: 1.748
0007903: 14300: Database: CDataHolder::Load: Field: Artist; Files: 4846; Pointer bytes: 39056; Data bytes: 16488; Elapsed ms: 0.891
0007904: 14300: Database: CDataHolder::Load: Field: Album; Files: 4846; Pointer bytes: 39056; Data bytes: 16488; Elapsed ms: 0.958
0007905: 14300: Database: CDataHolder::Load: Field: Track #; Files: 4846; Pointer bytes: 9694; Data bytes: 16488; Elapsed ms: 0.887
0007907: 14300: Database: CDataHolder::Load: Field: Genre; Files: 4846; Pointer bytes: 9982; Data bytes: 16488; Elapsed ms: 0.871
0007924: 14300: Database: CMediaInfoArraySort::Sort: Files: 4829; Elapsed ms: 26.120
0007925: 14300: Import: GetMissingAndUpdateFiles: Looping 4829 files
0007936: 14300: Database: CDataHolder::Load: Field: Library Server URL; Files: 4846; Pointer bytes: 0; Data bytes: 72; Elapsed ms: 0.364
0007968: 14300: Database: CDataHolder::Load: Field: Date Modified; Files: 4846; Pointer bytes: 39056; Data bytes: 16488; Elapsed ms: 1.026
0007969: 14300: Database: CDataHolder::Load: Field: File Size; Files: 4846; Pointer bytes: 39056; Data bytes: 32904; Elapsed ms: 1.048

It's apparently checking if the files still exist. It doesn't import any new files, but checks existing ones.
Looks like that library Cache file still contains references to the files, even though you deleted them from the library.
Try this:
- backup your library, close MC
- delete that Cache file: C:\Users\Rolf\AppData\Roaming\J River\Media Center 27\Library\Cache\File List Sort\Index.dat
- test to check if the issue is gone

There's 1 or 2 other files in that folder, you can likely delete them as well.
If the problem persists and that Cache file is rebuilt, then it means those file references still exist somewhere in one of the Library files.

Only someone from MC team can tell us what exactly this GetMissingAndUpdateFiles thread is doing.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 05:53:03 am

This is another small database. I created it from scratch a few weeks ago using MC25. All thumbnails created.
The files to be checked are referred to in the DB. But I didn't instruct MC to scan them again.

When opened with with MC27, background scanning starts immediately.

No background activity is logged with MC25.

With the larger database, this scanning has a further influence. It wakes sleeping HDDs. And MC has a problem when multiple drives need to boot at the same time. MC freezes and 'application does not respond! Quit or wait? ' will be shown.

Title: Re: MC scans files which are not in the DB
Post by: zybex on November 24, 2020, 06:08:03 am

I assume "Run auto-import in background" is OFF?

When you open a View/List with a playlist, I think MC will try to access each file to see if it's missing so that it updates the thumbnail with an indication of missing file. Not sure if this behavior can be disabled.

You can test by having it startup on a small playlist instead of "All Files" or similar, and see if it still scans all 4000+ files or just those ones.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 06:36:59 am

Quote from: zybex on November 24, 2020, 06:08:03 am

When you open a View/List with a playlist, I think MC will try to access each file to see if it's missing so that it updates the thumbnail with an indication of missing file. Not sure if this behavior can be disabled.

Yes this was the problem. It is called 'Display missing file image in lists (slow on network drives'. This was off in MC25 but has been automatically enabled in MC27.

The question that is still open: Why does the scan process (if activated) access files that are NOT in the DB?

Title: Re: MC scans files which are not in the DB
Post by: zybex on November 24, 2020, 06:52:41 am

Nice :) I also disabled that option now, lets see if the NAS stays sleeping more often. Also removed the isMissing() check from the bottom toolbar/Status line.

It's likely that its still scanning the Removed items database. Make sure you restart the Media Server as well.

On my Import settings (for each import folder), I uncheck the "Ignore files previously removed from library" - I *think* this causes MC to not add files to that Removed database - mine is always empty after MC restarts. Of course, it also means that you can't have files in your NAS folders that you don't want MC to import... if you delete from MC but not from the folder, it will just be re-imported next time. I'm good with that.

Title: Re: MC scans files which are not in the DB
Post by: glynor on November 24, 2020, 08:13:36 am

Quote from: zybex on November 24, 2020, 06:52:41 am

On my Import settings (for each import folder), I uncheck the "Ignore files previously removed from library" - I *think* this causes MC to not add files to that Removed database - mine is always empty after MC restarts.

That is correct.

Quote from: rolf_eigenheer on November 24, 2020, 06:36:59 am

Yes this was the problem. It is called 'Display missing file image in lists (slow on network drives'. This was off in MC25 but has been automatically enabled in MC27.

I mentioned that option in your other DB Performance thread. If you use Network Storage for your Media Library, this option is the single thing that has the largest performance benefit. What it does is checks any files visible in any view (or near-visible, it checks around the edges too) to see if they exist on disk so that it can put the missing from disk "anti" icon over them if they're missing.

That's slow on any setup where the media file storage is slow, especially Network disks (but can apply to external USB disks and whatnot too) because it has to actually check each and every file every time a view is displayed (or scrolled).

It defaults on, so if you had it off in MC25, it was because you had, at some point, manually turned it off. And, I guess, never restored the Settings when backing up and restoring to MC27.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 08:20:24 am

Good advice. I'll turn off "Ignore  files previously removed from library" too.

I'd rather get an import filter exactly like the filters used in views. The tags should be read from files, but before adding to the database the filtes should apply. So we could filter on size,  substrings in names or whatever. This would ensure that neither thumbnails nor DB entries would be generetad for files we want to skip.

Title: Re: MC scans files which are not in the DB
Post by: rolf_eigenheer on November 24, 2020, 08:23:29 am

Quote from: glynor on November 24, 2020, 08:13:36 am

I mentioned that option in your other DB Performance thread. If you use Network Storage for your Media Library, this option is the single thing that has the largest performance benefit. What it does is checks any files visible in any view (or near-visible, it checks around the edges too) to see if they exist on disk so that it can put the missing from disk "anti" icon over them if they're missing.

Sorry! I didn't check, because I already switched off that option. I was not aware that it comes back with every update. 

Title: Re: MC scans files which are not in the DB
Post by: RoderickGI on November 24, 2020, 02:37:10 pm

Quote from: zybex on November 24, 2020, 06:52:41 am

On my Import settings (for each import folder), I uncheck the "Ignore files previously removed from library" - I *think* this causes MC to not add files to that Removed database - mine is always empty after MC restarts.

Quote from: glynor on November 24, 2020, 08:13:36 am

That is correct.

Actually no. MC still adds files removed from the Library to the Removed Database. It is just that you two have "Ignore files previously removed from library" set so MC re-imports any files you delete next time Auto Import runs, if those files still exist, and the files are removed from the Removed Database when they are re-imported.

If you have "Fix broken links" in Auto Import set to "No", then files that no longer exist will remain listed in MC. When you delete those missing files from the Library they will be added to the Removed Database, but when Auto Import runs with or without "Ignore files previously removed from library" set they will be removed from the Removed Database, and of course, aren't imported back into the Library.

We seem to have covered stuff that was discussed before, but that is okay. It is reinforcing the issues with some settings.

1. Auto Import turned off so it doesn't automatically run in the background, but only when manually run, stops MC scanning for new files regularly.
2. Turning off "Display missing file image in lists (slow on network drives)" and don't use IsMissing() anywhere it is always displayed (Status Bar), or unnecessarily in Views, without understanding the consequences.
3. Decide how to manage files removed from the Library when they aren't removed from disk. Be aware of the consequences of "Ignore files previously removed from library", particularly in combination with "Fix Broken Links" settings.
4. When restoring Library Backups from earlier versions, restore Settings as well, if you want MC to behave in the same way as the earlier version. But also check the MC Options to be sure all settings are as you wish. Sometimes with functional changes in later versions, MC will set some Options back to defaults, so that no invalid Options are left in place, or for perceived performance or usability of MC.

Title: Re: MC scans files which are not in the DB
Post by: glynor on November 24, 2020, 03:27:07 pm

Quote from: RoderickGI on November 24, 2020, 02:37:10 pm

Actually no. MC still adds files removed from the Library to the Removed Database. It is just that you two have "Ignore files previously removed from library" set so MC re-imports any files you delete next time Auto Import runs, if those files still exist, and the files are removed from the Removed Database when they are re-imported.

If you have "Fix broken links" in Auto Import set to "No", then files that no longer exist will remain listed in MC. When you delete those missing files from the Library they will be added to the Removed Database, but when Auto Import runs with or without "Ignore files previously removed from library" set they will be removed from the Removed Database, and of course, aren't imported back into the Library.

You're absolutely right. Sorry. I actually thought that might have been wrong after I posted it, but never came back here till now.