INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: What exactly is happening during "Adding files to library..." during import?  (Read 2767 times)

Negasonic

  • World Citizen
  • ***
  • Posts: 115

Something that has been "bothering" me for 2-3 years now, so I thought it was about time to ask :D

After adding an album to the library, I click "Run Auto Import Now" and the "Importing Media" window pop up. In this window it says first "Searching for files..." and I can see it quickly browse through the folders.

After this it says "Adding files to library..." where it starts counting from 0 to somewhere between 1500-1600 every single time, no matter if I have added 0 or hundreds of files since last import. Sometimes this counting can take quite a bit of time and other times it goes pretty quickly.

What is it counting or "adding" at this moment? Any idea?

Just like to add that import works otherwise, I'm just curios about that "adding" part, because I have certainly not added 1500+ files each and every time.


Edit:
ANSWER: CUE files pointing to a single file album + imported playlists (M3U etc) are being re-checked every time import runs and being added to the counter while "Adding files to library..." is displayed
Logged

syndromeofadown

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 805

Just a guess but maybe you have some read only files. Using windows explorer make sure all the files in your import folder are not checked "read only".
Logged

MikeO

  • Citizen of the Universe
  • *****
  • Posts: 789

I always assumed it was image files in album folders

Not too sure why

Would be nice to know

Mike
Logged

Negasonic

  • World Citizen
  • ***
  • Posts: 115

Just a guess but maybe you have some read only files. Using windows explorer make sure all the files in your import folder are not checked "read only".

I checked this, but the counting during "Adding files to library..." up to 1615 (which seems to be the current number) remains, although it went much, much faster now after un-ticking the Read-Only box. Only a few seconds compared to, sometimes, over a minute before. So, at least that's a plus! :)


I always assumed it was image files in album folders

Not too sure why

Would be nice to know

Mike


I think you are right. Still not sure what is going on, though.

I just did a quick test now by adding 5 random .jpg's into an album folder and during the first import, it counted up to 1620 (compared to 1615).
Then I hit Import again and now it was back to 1615 again.
Same thing third time.
Restarted JRiver. Import again - still 1615.
Removed the 5 image files - import - still 1615.

Logged

Sauzee

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 714

Thanks for the detective work. I wondered about that too.
Logged

Negasonic

  • World Citizen
  • ***
  • Posts: 115

I have a feeling that those 1615 files it continually "adds" each and every time I hit Import, are not part of my library or, at some point, was part of my library but no longer is... or something  ?

I guess that during the "adding files to library..." part it is supposed to only add newly added files, so if I add 10 music files + 5 images it should only count from 1 to 15. And if nothing has been added it should not count anything at all.

Currently all my files are on 2 internal drives. I used to have an external hard drive with files that were part of the library, a drive that was not always connected to the computer. I wonder if the problem (if it is a problem) started during that time. That when the drive was not connected JRiver would still load up the info about those files (but with a red cross on the thumbnails). This drive is no longer part of my library. About a year ago it crashed and could not be used again.

I don't remember what I did to the library after that HD was removed for good. I have removed it from the Import configuration, but apart from that I am not sure what I did. The files do at least not show up in the library any more.

Does anyone know if there is a way to check if there is some old "crap" lingering in the database in some way? To find out what those 1615 files it keeps picking up during import are?

Appreciate all the help
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186

Have a look in the "Removed Database" in MC.

It could be that when you removed the external drive, you deleted all the files that were on it from your library, and MC is still checking that they aren't to be imported. Not sure, because if the files are physically gone, then I think usually they disappear from the removed database. But if you have the files still, but now on an internal drive, MC may still have them in the Removed Database.

If the files are in the Removed Database and no longer need to be, delete the record from that database.

Search the forum for "Removed Database" if you don't know how to look at it.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

~OHM~

  • Citizen of the Universe
  • *****
  • Posts: 1825
  • "I Don't Play The Music The Music Plays Me"

Search the forum for "Removed Database" if you don't know how to look at it.
I did this search, all I came up with was this post!
Logged
“I've Reached A Turning Point In My Life. I Now Realize I Have More Yesterdays Then Tomorrows”

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71438
  • Where did I put my teeth?

Search from the main index board or use the Advanced search.  The default search just searches the current location.
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

~OHM~

  • Citizen of the Universe
  • *****
  • Posts: 1825
  • "I Don't Play The Music The Music Plays Me"

Search from the main index board or use the Advanced search.  The default search just searches the current location.
Oh DAH me...thanks Jim
Logged
“I've Reached A Turning Point In My Life. I Now Realize I Have More Yesterdays Then Tomorrows”

Negasonic

  • World Citizen
  • ***
  • Posts: 115

Have a look in the "Removed Database" in MC.


Thanks for the tip! There was nothing there, though. Whenever I have deleted something from the library I also deleted from disc. The files from that external HD were never copied to any of the internal HDs, so they are totally gone (well, I still have the discs).

Not sure what else can be done? I'm so curious at this point about what those 1615 mysterious files could be, that I can't let it go  :P 
Logged

TreeNebula

  • Member
  • *
  • Posts: 3

You've asked the question that has been the bane of my jriver experience for YEARS.

Just now I went into import > configure auto import > click edit > under the list of audio files, deselect CUE files.

Now it no longer goes through the agonizing process looking at CUE files (212 for me). What will this affect? If you have FLAC images (they need the CUE files) those albums might disappear from your library. I recommend using CUETools to split your FLAC images.
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608

If you have FLAC images (they need the CUE files) those albums might disappear from your library.

It won't remove any already added to your Library. It will just stop importing new ones.

If it is caused by CUE playlists in the import paths (or other playlist types), you may be able to stop it re-importing them, without disabling them entirely, by unchecking Update for External Changes in the Auto-Import Options, under the Tasks section. When this option is on, it forces MC to check files on disk and re-sync tag metadata from the files that have changed back into MC. I've definitely seen this option cause long imports from updating sets of files referenced in Playlist type files (CUEs and similar) before.

If you aren't typically changing files regularly with separate (external) applications, then it won't hurt to turn that off. And, if you do change any of them and need MC to sync the changes, you can always select them manually and do Update Library (from Tags) to force it for those files.
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

Negasonic

  • World Citizen
  • ***
  • Posts: 115

You've asked the question that has been the bane of my jriver experience for YEARS.

Just now I went into import > configure auto import > click edit > under the list of audio files, deselect CUE files.

Now it no longer goes through the agonizing process looking at CUE files (212 for me). What will this affect? If you have FLAC images (they need the CUE files) those albums might disappear from your library. I recommend using CUETools to split your FLAC images.


Thanks for your post. I am pretty sure now it is .cue files causing my issue. Using Windows File Explorer and typing ext:.cue in the search field of my music folder gave me a result that was very near 1615.

I have not done any further investigations so far. The majority of my cue files are related to single file albums (the tracks are not split into separate files, but bulked into one file). Some others exist in folders where the tracks are split, though. 

Searching for .cue files, I found a quote from this thread: "Existing files re-importing on every import pass - LOG EXTRACT INCLUDED"
https://yabb.jriver.com/interact/index.php/topic,114849.0.html

CUE files and SACD ISO files import like this. For some reason Media Center always thinks that these files are new every time import runs.
This is problematic if you have files that you wish to be removed from the library, as Media Center re-imports them every time auto-import runs.

Is this correct? And if so, why?
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608

Is this correct? And if so, why?

I could be wrong, but I believe this is only "correct" if the Update for External Changes option I discussed above is enabled. Even then, they aren't actually re-imported, but it does have to re-scan them each time Auto-Import runs, to determine if they've changed externally.

Did you try disabling that as I suggested?
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

Negasonic

  • World Citizen
  • ***
  • Posts: 115

I could be wrong, but I believe this is only "correct" if the Update for External Changes option I discussed above is enabled. Even then, they aren't actually re-imported, but it does have to re-scan them each time Auto-Import runs, to determine if they've changed externally.

Did you try disabling that as I suggested?

Sorry, forgot to mention that I did try that. It made no difference, though, it was still counting through the cue files with Update for External Changes deselected.
Logged

Negasonic

  • World Citizen
  • ***
  • Posts: 115

So...

The files being added to the library but not really being added to the library during import are single files (containing multiple tracks) with a corresponding cue-file in the same folder. Deleting 1 album of this kind from the library and hard drive made (in my situation) the number it counts go down from 1615 to 1614.

What happens during "Adding files to library..." is that MC finds the cue file and the single music file and - instead of adding these two particular files to the library - it instead adds the tracks that the cue file is pointing to that are contained inside the single music file. As some kind of "virtual files" for each track?

Next time (and next next etc) you use import, MC will once again find those cue and single files (Regardless if you have "Update for External Changes" on or off) since they were never actually imported into the library in the first place. Only the "virtual track files" were imported. So it keeps "adding" these files every single time you start import only to realize that it is not supposed to add those files.

Or something.
?
 
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608

Next time (and next next etc) you use import, MC will once again find those cue and single files (Regardless if you have "Update for External Changes" on or off) since they were never actually imported into the library in the first place.

This part of your statement is not correct. The tracks from the cue file compound document were added to the Library.

I retested this, and you are correct that MC does have to re-scan any Playlist type files in the import paths at each import. CUE files are playlists, and so they must be re-read at each import to check to see if the contents have changed. This makes sense, as Playlists aren't "files" per-say, and their contents could change from import to import. For example: Imagine you have an external application that generates new dynamic playlists once per day, and you want MC to keep syncing itself with this new content. It can't "know" the contents of the playlist file are identical without reading the file, and parsing each entry in the file to check to see if there are any changes.

That's what importing a CUE file means. Parsing the file and adding track entries with the metadata from the CUE file to define the tracks in the database.

If you don't like it, and you want to keep the file format of your rips untouched, the solution is simple:
1. Make an Imports folder specifically for new CUE rip.
2. Have MC's Auto-Import monitor only this folder (or, at least, monitor only it for CUE files).
3. Once imported, use Rename, Move, and Copy Files in MC to move the files out to their long-term storage location. This folder won't be watched at all, or if so, will have importing CUE files disabled.
4. Done. Drink a beer and enjoy your future fast imports again.
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

Negasonic

  • World Citizen
  • ***
  • Posts: 115

Thanks for your post and all the info and help!


This part of your statement is not correct. The tracks from the cue file compound document were added to the Library.

Well, that's actually what I wrote, in the sentence that came after your quote, that only the tracks or what I called them "virtual track files" were added. The actual cue file and the actual music file (the bulk file/disc image file) are not imported. If I do a search now for cue files in my library it will come up with zero.


The main confusion for me (and maybe others?) came from these playlists (cue files) being re-checked and MC adding new files to the library being lumped together in the same step ("Adding files to the library...") during the import. The tracks from the cue files have already been added to the library.

I think it would have been more clear if perhaps when doing import, MC and the "Importing Media" window would check and display info in this order:

1. "Searching for files..."  (MC browse the folders for something new)
2. (NEW!) "Checking/Adding Playlists and Cue files..." or something (The counter starts ticking when MC checks through playlists of all kind - in my case this is where it would count to 1614)
3. "Adding files to the library..." (A new counter starts showing only newly added files to the library)

I would be perfectly ok with that. And maybe clearer for other users. I mainly just wanted to know why MC seemed to be adding so many new files every time I started Import.
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608

The actual cue file and the actual music file (the bulk file/disc image file) are not imported. If I do a search now for cue files in my library it will come up with zero.

Right. Because the CUE files are playlists, not media files. They define the "virtual assets" in the database. Both the CUE and bulk media files are imported, they are just represented in the database by multiple media files. This is no different than importing a M3U playlist file, except that the CUE file points to different sections of the same media file (typically) and a M3U file points to multiple discreet files on disc.

If you have M3U files in your import path, and the Playlist file type grouping fully enabled, MC re-scans these at each import too. I thought originally that you could disable this by disabling the Update for External Changes option, but I was wrong. I re-tested before I posted the above, and it re-scans all playlist file types each time. This makes sense as they are, by definition NOT media files, but playlists. When you import other playlist type files, those also don't show up "directly" in the Library. They are converted to MC's type of database-Playlist and put under Playlists in the Tree.

1. "Searching for files..."  (MC browse the folders for something new)
2. (NEW!) "Checking/Adding Playlists and Cue files..." or something (The counter starts ticking when MC checks through playlists of all kind - in my case this is where it would count to 1614)
3. "Adding files to the library..." (A new counter starts showing only newly added files to the library)

Sure. That would be, logically, more understandable. However (I'm not positive, but from the work I've done on MCFileIngester, I believe that) architecturally, MC's Import engine doesn't distinguish between #2 and #3. It basically does this (for any full Auto-Import run, background runs work differently):

1. Filters the list of all files in a Watched Directory against enabled File Types for the Import.
2. Checks this list against the current files in the Database.
3. Any completely new media files get added to the temporary "Import Queue" list.
4. If Update for External Changes is enabled, then it also adds any files that have a date modified date newer than the last database update timestamp for the file in question.
5. Any playlist format files it discovers are added wholesale to the Import Queue list. It has to do it this way because the search engine does not read files from disk (just looks a filesystem metadata). To know if there are new files in Playlists, you can't just look at the filesystem metadata (mostly the filenames on disk), you have to actually open and read each file. Reading files is done by the Import engine, not the search part of the engine.
6. All of the above is the "Searching for files" part of the process. That search is fast precisely because it doesn't read any files off of disk at all, just filesystem metadata. When the search is done, it hands the Import Queue list it built to the "Import Engine".
7. The Import Engine reads each file in the Queue and each file is either imported fresh (if it never existed previously in the database) or, if it exists already, the metadata in the entry for the file is refreshed to match the on-disc data.

You're looking at it like "importing" is a one-time thing, and that updating the Library is a discreet operation. While this makes sense logically, I think technically those two things are done by the same code, and are, as far as MC's components are concerned, identical operations. It is reading the file from disc and updating the database. Whether it is adding a new row to the database or updating an existing row is pretty irrelevant.
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

TreeNebula

  • Member
  • *
  • Posts: 3

Not sure if we're past this, but I think it has to do with all cue files, not just flac images (single file flac albums). I painstakingly went through my collection looking for flac images and split them if any were discovered.
Logged

Negasonic

  • World Citizen
  • ***
  • Posts: 115

Not sure if we're past this, but I think it has to do with all cue files, not just flac images (single file flac albums). I painstakingly went through my collection looking for flac images and split them if any were discovered.

Yes, it is cue files (also Playlists (m3u etc) but have not checked personally since I don't have any), but only cue files pointing to a single file album (no matter if it is flac or mp3 or something else). Cue files with separate track files are not affected, at least not in my experience. I deleted all those files awhile ago and it did not alter the counter in any way during the "Adding files to library..." procedure.

Cue files with separate track files gets imported into the database. You can search, find and delete these cue files from within MC. I have not found a way to find the cue files that point to a single file album from withing MC, which begs the question: Is there a way to set up a list/view in MC to display only albums that consists of a cue file pointing to a single file flac/mp3 etc?

You're looking at it like "importing" is a one-time thing, and that updating the Library is a discreet operation. While this makes sense logically, I think technically those two things are done by the same code, and are, as far as MC's components are concerned, identical operations. It is reading the file from disc and updating the database. Whether it is adding a new row to the database or updating an existing row is pretty irrelevant.

Once again, thanks for all your input. I've learned a lot through this thread.

Would it be possible, do you think, to add a "Dear MC, during the Import procedure, please leave my cue files alone, after they have been imported the first time. No need to re-check them again, no more stalking, no more harassing. I have not made any changes to them and I never will, I promise" - type of setting in the Configure Auto-Import window? Worded exactly like that, of course  :P

The "Update for external changes" option seems like it should be the right thing to handle this, but as we have tested, it doesn't. But maybe a refined version of it, only for cue files could be added? I understand the need to re-scan regular playlists where the content may be a bit more fluctuating from time to time. But cue files pointing to a single file album?

And another thing,  deselecting cue files in the import settings has been mentioned a couple of times in this thread and... I actually tried this a couple of days ago and I am not sure what happened but it kind of messed up the library. I did not spend much time investigating, I immediately reloaded the library from a previous save. At a quick glance It seemed like all single file + cue albums were missing track #1, for instance. When it skipped the cue file during import, it looked like it imported the bulk files instead, or something... I don't remember. Just... be cautious if you do this and remember to do a library backup before any changes.   
Logged
Pages: [1]   Go Up