Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[hells]

Hey all! I tried searching about importing, but it brings up results from ~2002, so thought I'd post a thread.

My issue is setting up autoimport works great the first time, but it seems as it automatically does its thing, the media starts duplicating. We're talking identical paths (local not remote shares) filenames, etc. It just keeps accumulating entries. Every album will get the same number of additional duplicates too. So each run is 1 duplicate track for every track on every album.

Is there some knob I missed to stop this? I have fix broken links on "yes" just in case that was the issue, but alas not solving the problem.

Thanks!

EDIT: Perhaps this should be moved to the Linux forum. At first I assumed it was my Windows client possibly creating these entries, but my docker container that hosts the server is actually creating 20 or more copies for every track in my library, on every import. I'm not sure if server mode is important in this, but that's all it is functioning as.

[Cmely]

Hi,

I'm replying to this topic because I get the (almost) same problem.
Maybe Hells you finally managed to find a solution to that ?

I'm running MC27 on a Linux Ubuntu 20 PC, wich I use as a server.

Each time I run MC, it will import again the same file (exact same path, exact same file). What is weird in my situation is that only few albums (always the same) are re-imported. They are the 6 or 7 last imported music albums.

If you have any idea on how to solve this weird behavior, I'll be happy to hear it

[hells]

Unfortunately I did not, I tried many things.

It may be due to permissions or something weird with mtime/atime and docker, but I couldn't figure it out.

[zybex]

Can you please post the full path to one of the affected files? Perhaps it includes a reserved character that MC is replacing on import.

[Cmely]

\var\snap\nextcloud\common\nextcloud\data\imserveur\files\Musique\Divers\Leprous\Pitfalls (Album)\01 Below.flac

At first, I thought it was because of "/" or "\" character, so re did the import and the "search and replace" carrefully for a correct path. But it didn't change anything.

Like Hells, I suspect a weird problem of permissions...but I can't figure out what. Albums and files that are re-imported have the same permissions (to my knowledge, I'm not a great Linux user) than other albums that are not re-imported 

[zybex]

Are you using NextCloud to sync your files to some other PC/Server/NAS? If so, try disabling it and then import the folder 2 or 3 times to see if the issue is still there.

File sync services can cause issues because they modify and virtualize the files they handle. If you are syncing 2 or more PCs and importing the same files in both of them, the tags added to the file during import on one side cause the file to be modified and re-synchronized to the other side. If you're using it to sync 2 local folders and then importing both folders, the same will occur.

[wer]

If the filespec that MC accesses is truly identical, MC should consider them the same file.

Please post a screenshot in MC with the file list of the view visible, and with the entire contents of the [Filename] field visible as a column, for the affected files. Make sure the column is wide enough that the entire contents is visible.

[Cmely]

Here is the screenshot.
The filenames are exactly the same...and when I use the command "Locate on explorer" on both files, I get the same result...

[zybex]

If the filespec that MC accesses is truly identical, MC should consider them the same file.

I was thinking of 2 special cases:

1. PC A imports the file from NAS1, which adds tags to it; this causes the file to be re-synced to NAS2. If there's another MC on the other side, it will see that as a modified file and re-import it, possibly modifying it again and triggering a loop between the 2 PCs (though in this case I don't think there would be duplicate entries in MC)

2. Even with 1 PC only, if sync virtualization is enabled the file's inode number changes after the file is uploaded (a new empty sparse file is written in its place). This may cause MC to see it as a distinct file and reimport it; the reimport causes the file to be materialized, and the loop repeats.

[wer]

It was my understanding that MC checked the filename to see if the file was already in the DB or not. If that was the case, it shouldn't be able to import the file twice, changed inode or not, NAS or not, cloud or not, if the [Filename] is identical.

The screenshot appears to show identical filenames. I wish that screenshot were of the whole window, so we could see if there are stacks, or broken links, or particles, or anything else.  Stacks or particles can produce the screenshot shown, for as much as it shows.

There have been instances in the past where people had unusual characters like &nbsp in their filename, which were visually indistinguishable from regular characters, but MC considered them different and imported both files.  Something like that could be involved here.

[Cmely]

Are you using NextCloud to sync your files to some other PC/Server/NAS? If so, try disabling it and then import the folder 2 or 3 times to see if the issue is still there.

Not exactly. My setup is this one :

- One Windows 10 PC (WPC) wich is my working computer with a 'Music' folder where I drop all the music I want to add to Media Center.
- One Linux Ubuntu 20 PC (LPC) which I use as a Nextcloud server.
- I don't use Nextcloud sync (it's not working very well)
- I use Syncback Pro to do a one-way sync of my music from WPC to LPC : when Syncback detect a new file on WPC it copies it on LPC. Theorically, once the file has been copied, sync never occurs ever again for this file.
- I play my music on WPC from Media Center Media Server on LPC.

[Cmely]

The screenshot appears to show identical filenames. I wish that screenshot were of the whole window, so we could see if there are stacks, or broken links, or particles, or anything else.  Stacks or particles can produce the screenshot shown, for as much as it shows.

Wer, first thank you for helping. What would you like to see ? The path you see on the screenshot is the complete path of the file (at list as it appears on MC).
What are 'stacks' and 'particles' ?

[zybex]

Are there any symlinked folders on that path? I'm not sure how MC handles them on Linux. It's possible that the same files are being imported via two distinct paths, but then MC is showing the same canonical (real) path on the UI.

you can check how many files you get with something like:

Code: [Select]

find /var/snap/nextcloud -name "04 Ocean.flac" -print

[JimH]

Please see the wiki.

[wer]

Cmely, I'm glad to try and help. Your problem is a strange one. if you don't know what they are, you are probably not using them. But we have to check possibilities because what you are reporting should be impossible.  Google jriver stacks and jriver particles to learn what they are.

Their presence is usually indicated by icons in he far left columns in the view.  Please just show the whole window, including the leftmost columns.  You can drag the filename column so it is one of the leftmost columns so it stays on the screen. We need to be able to correlate the contents of that field with the icons in he leftmost colums.

[wer]

I'm not sure how MC handles them on Linux.

MC uses the symlink path instead of the resolved path, for the exact reason I described before. So such a file will show in the library twice, but with two different filename fields.

[Cmely]

Cmely, I'm glad to try and help. Your problem is a strange one. if you don't know what they are, you are probably not using them. But we have to check possibilities because what you are reporting should be impossible.  Google jriver stacks and jriver particles to learn what they are.

 
This is the case : I've never ever used neither stacks or particles. I've never even noticed those options of grouping.
I think we can cross out this hypothesis.

[Scobie]

Certainly a weird issue.

Does this happen when you look at your entire library when you run an import, or does it also happen when you drill down to a single folder and import that?

Also what are your auto import settings? Do you have "Update for External Changes" set? And do you have a virus scan scheduled on your music folder(s)?

[Cmely]

Certainly a weird issue.

Does this happen when you look at your entire library when you run an import, or does it also happen when you drill down to a single folder and import that?

Also what are your auto import settings? Do you have "Update for External Changes" set? And do you have a virus scan scheduled on your music folder(s)?

Yes, I agree, it's very weird 
I'll try to make more tests this weekend to see if I can narrow donw some hypothesis...

To answer to your questions, I only use "autoimport" and there is only one folder (always the same) which is looked at. Media Center is always active. I don't know about "update for external changes", I'll look for it tonight.

I don't have any virus scan scheduled on my music folder.

[leezer3]

What filesystem?

I'm wondering if it's something more esoteric (BTRFS?) and MC isn't liking something like snapshotting.

[Cmely]

I (think) the filesystem is Linux ext4.

I've been running more tests and tries...and I've got really weird result.

First of all, I've uncheck "Update for external changes". I also played (a little) with folder and files permissions.
I also checked the "do not reimport deleted files".

So know, the result I get each time I add files in my "autoimport" folder are :
- Only few albums are affected by this problem. It's always the same 6/7 albums and sometimes one of those recently imported...but not all the time...
- I get sometime up to 8/9 duplicate of each song for those albums
- After a certain amount of time (several minutes), these duplicates resolve themselves and disappear.
- In some of those 7 albums, all files are duplicated, in some of those files some files are duplicated.
- Whatever the result can be, the "imported date" of those 7 albums change each time to the imported date of the last album imported.

... ?  ? ?

I really don't know what to think...(I've added few screenshot)

[zybex]

Isn't this the normal behavior when there are playlist files such as .m3u on the folder? MC imports both the audio files and the playlist references (creating duplicates), and then at the end of the import process resolves the duplicates.

I'm not sure about this since I don't use MC for audio.

[Cmely]

Isn't this the normal behavior when there are playlist files such as .m3u on the folder? MC imports both the audio files and the playlist references (creating duplicates), and then at the end of the import process resolves the duplicates.

I don't have any .m3u files in my folders...

[zybex]

Are there any other files apart from the .flac files and album art? Another common playlist format (sort of) is .cue.

[wer]

Show us the contents of the filesystem.  Give us an ls -l in a directory with the duplicates, and/or the output of the find command Zybex asked for previously.

[zybex]

"ls -la" (single dir) or "ls -laRF" (recursive tree) to include hidden files too.

[wer]

We don't need recursive I don't think. One directory where MC shows duplicates would be a good start. But redirect it to a file and post the file, instead of a screenshot.

[Cmely]

Are there any other files apart from the .flac files and album art? Another common playlist format (sort of) is .cue.

No.

I've checked more seriously for m3u files and I found some, but none of them are related to the 7 albums still duplicated.
I've deleted them (the m3u files) just in case.

[Cmely]

(Just wanted you to know, guys, that I really appreciate the effort you're putting into trying to resolve this mystery. Thank you)

This is the result of ls-l in one folder :

Code: [Select]

Total 504524

-rw-r--r-- 1 root root 43413289 janv. 17 21:33 '01 Reggae Samba (Live).flac'

-rw-r--r-- 1 root root 72367114 janv. 17 21:33 '02 Bahia (Live).flac'

-rw-r--r-- 1 root root 39763324 janv. 17 21:33 '03 Posto De Gasolinha (Live).flac'

-rw-r--r-- 1 root root 38432010 janv. 17 21:33 '04 Copacabana (Live).flac'

-rw-r--r-- 1 root root 18847152 janv. 17 21:33 '05 Rio De Janeiro (Live).flac'

-rw-r--r-- 1 root root 69454964 janv. 17 21:34 '06 Sluneční Dům (Live).flac'

-rw-r--r-- 1 root root 52817791 janv. 17 21:34 '07 Afropop (Live).flac'

-rw-r--r-- 1 root root 18773270 janv. 17 21:34 '08 Quaquanco (Live).flac'

-rw-r--r-- 1 root root 52488212 janv. 17 21:34 '09 Rumba Samba (Live).flac'

-rw-r--r-- 1 root root 59753892 janv. 17 21:35 '10 Acid Samba (Live).flac'

-rw-r--r-- 1 root root 50117078 janv. 17 21:35 '11 Final Song Samba Praga (Live).flac'

-rw-r--r-- 1 root root   387189 janv. 17 21:35  cover.jpg
And I've attached the screenshot of the result in MC.

[mwillems]

Cmely, is this the same library that you ported over from Windows that we discussed a few months back?  (See https://yabb.jriver.com/interact/index.php/topic,127962.0.html)

If so did you remember to change your platform.jmd file as discussed in one of the threads I linked?  If you did not change the platform.jmd file it can cause this exact issue.

See this thread from last month for someone else seeing selective duplicates on auto-import where platform.jmd was the issue: https://yabb.jriver.com/interact/index.php/topic,127935.msg887584.html#msg887584

I've quoted some relevant info from that thread here:

Can you post the content of your ~/.jriver/Media Center 27/Library/Platform.jmd file?  If it's anything other than "Linux" that would be a good sign that you're seeing something similar to what I saw.  For me fixing the platform.jmd and then deleting and reentering all of my auto-import rules (which had the wrong path structure) fixed the duplicate imports, but I had to do some manual clean up of duplicates afterwards.

There have been a few longer threads about this also:

https://yabb.jriver.com/interact/index.php/topic,117593
https://yabb.jriver.com/interact/index.php/topic,123748.0.html

[Cmely]

Cmely, is this the same library that you ported over from Windows that we discussed a few months back?  (See https://yabb.jriver.com/interact/index.php/topic,127962.0.html)

It is absolutely the same library, yes. Thank you for reminding me this thread...

If so did you remember to change your platform.jmd file as discussed in one of the threads I linked?  If you did not change the platform.jmd file it can cause this exact issue.

I did absolutely not remember anything... 

But you did...and I'm glad to tell you your reminder solved my duplicate problem !!!

Many thumbs up Mwillems and thank you very much : I did follow the advices mentionned on the thread you redirect me to. So I've changed my "platform.jmd" file and set the "fix broken links" option to 'yes'.

Now import is working perfectly fine : no duplicates, no screwed up 'imported date' values and all the files that were previously duplicated are gone.

Merci beaucoup.

[mwillems]

Excellent!  I love it when a plan comes together!

[wer]

The penguin shoots and scores! 

[zybex]

Kudos!