Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[bblue]

When you use Remove Duplicates, what exactly is matched to determine a duplicate?  All tags?  File characteristics?

If it's tags, are there any tags that are NOT evaluated?

I can't seem to find anything authoritative on the subject.

[MrC]

In a Playlist, a duplicate would be the same Filename - this is MC's knowledge of the file's path.

More specifically, it may be the FileKey(), the unique entry in MC's database.  From a user's perspective, you can focus on Filename for any MC entry that has a physical presence on disk.

[tlin1688]

Remove Duplicates won't work if the supposed unique filekey not unique.

How could that be?

Well, things like you changed file folder name or drive letter, then you add that file again into the playlist.
if you change mind and rename the folder(or drive letter) back, and then use (Library Tools-->Rename Move Copy...) to modify your playlist to match. You then create a duplicated FileKey().

That's something it won't. Sometimes, "Remove Duplicates" will do too much, (BUG, that is)

When you are on some playlist, say, a list of 1000 movies, if you do a "search", and got 60 movies, and perform a "Remove Duplicates", maybe just accidentally, on the search result list. Oops, your playlist GONE!!!

Well, not entirely, all but the search result. Your playlist now has only 60 entries. 940 movies GONE.

If you want to try this at home, do it on a duplicated playlist, I mean, copy the playlist first

[TCube]

@ tlin1688

Edit/delete
.../...

What would be the purpose on building playlists on duplicated files and thereafter looking for way to remove duplicates within a playlist... please.
TC
p.s I never came across this case for myself.

Partitioning, restoring from a back-up, etc 

[bblue]

In a Playlist, a duplicate would be the same Filename - this is MC's knowledge of the file's path.

More specifically, it may be the FileKey(), the unique entry in MC's database.  From a user's perspective, you can focus on Filename for any MC entry that has a physical presence on disk.

Ok, that makes sense.  But how would duplicate playlist entries come about, then?  Simply by importing the same sub directory (say a whole album) again?

I'm seeing some strange duplications of particular albums here and there.  A couple of them I recognize as being renamed or in some way reorganized within their hierarchy, but others that were similarly modified do not show up as duplicated.  At least in the past it seemed to be pretty smart about this sort of thing, and if files or sub directories of your hierarchy were removed and re-established it was handled in a very clean fashion.  They'd first be deleted from the playlist/library and then on a pass from the library import scanner, would re-appear as new... or so it seemed, anyway.  And they'd show up in Recently Imported.  But a few months back Recently Imported seemed to just stop working.  New albums/directories added were never noticed unless you specifically dropped them into a playlist.  In this latest version .131 it seems to be hit and miss but I haven't found any pattern yet.

Has there been any new criteria for what constitutes an import?  I always thought it was automatic from the import scanner -- anything that is there now that it doesn't know about is auto-added and shows in Recently Imported, but now it doesn't seem consistently that way.  It was great to be able to pull up Recently Imported, go to Tags and select the playlists that each track or album should live in, and lastly uncheck Recently Imported as a playlist where the tracks also live.  That way I could update as I have time, regardless of how many actual albums have been added over xx weeks or months.  (and yes, I have the option not to clear on exit enabled)

Anyone seeing this or have any other info about it?

As it is now I have to keep a list of adds into my library and when ready for maintenance manually drop each of them in to my ALL Playlist folder.

Not fun.

--Bill

[MrC]

I think there are several things you're dealing with...

There are user created playlists, and there are "play"lists created / modified by MC.

A "playlist" is just a list of file references (or URLs, or other, but I'll ignore those for now).  A playlist may reference the exact same physical file one or more times.  How do multiple references get into a playlist?  A user adds them to the playlist.

The "strange duplications of particular albums" issues seems to be suggesting that you have inadvertently imported the same file set one or more times, where each time the file resided at a different file path then existing MC entries.  So if you move a file out from under MC, and then reimport this file from somewhere else, MC can have two entries for what you consider the same file.  But they were not the same file, because a file is defined by its file path location, not its contents.

You're also indicating that MC's auto-import doesn't appear to be adding files to Recently Imported.  This may be due to an issue some have reported.

MC's Auto Import can do a few things for you, for files under watched (auto-import) folders:

   - It can Fix broken links (i.e. the file pointed to by the Filename field).  MC will try to re-connect a moved/renamed file if it can, or when it can't, it can remove the file reference from the database (Library).

   - When you delete an entry from MC, it can (when Ignore files previously removed from library is enabled) move those references into a special Removed database.  This is how MC remembers you've deleted a file.  And MC will move this file from the Removed database back in your Main database if you ask MC to re-import it.  Or if you removed the entry from the Removed database manually, MC will import this file as if it were new.

I wouldn't rely on the Recently Imported playlist, since MC pretty much owns this.  Instead, create your own Smartlist that shows you the last N days (or weeks or whatever) of imported files, from most to least recent.

Also, I have posted in the beta forum a view to look at your recently imported files.  I'll post it if it is useful.

See also:

   http://wiki.jriver.com/index.php/Import

[MrC]

Also, I think the algorithm for Recently imported is something like:

An item remains in the playlist if it is imported within the last N days, or if it was one of M most recently imported items.  And Recently Imported is cleared/removed if the newest item is older than X days.

I don't know the values of N, M and X, but those could be discovered empirically.

[bblue]

I think there are several things you're dealing with...

There are user created playlists, and there are "play"lists created / modified by MC.

A "playlist" is just a list of file references (or URLs, or other, but I'll ignore those for now).  A playlist may reference the exact same physical file one or more times.  How do multiple references get into a playlist?  A user adds them to the playlist.

The "strange duplications of particular albums" issues seems to be suggesting that you have inadvertently imported the same file set one or more times, where each time the file resided at a different file path then existing MC entries.  So if you move a file out from under MC, and then reimport this file from somewhere else, MC can have two entries for what you consider the same file.  But they were not the same file, because a file is defined by its file path location, not its contents.

Right.  But in every case the duplicates refer to the exact same file and path.  In my music hierarchy, which is part of an all-media hierarchy on my library server, there are no duplicates and everything is in E:/FLAC/flacHD/Artist/Album Name/track#-track-title format.  The E: directory is network shared and any other system, including MC Clients, has a local directory E: from the server share and therefore an identical hierarchy.

You're also indicating that MC's auto-import doesn't appear to be adding files to Recently Imported.  This may be due to an issue some have reported.

MC's Auto Import can do a few things for you, for files under watched (auto-import) folders:

   - It can Fix broken links (i.e. the file pointed to by the Filename field).  MC will try to re-connect a moved/renamed file if it can, or when it can't, it can remove the file reference from the database (Library).

   - When you delete an entry from MC, it can (when Ignore files previously removed from library is enabled) move those references into a special Removed database.  This is how MC remembers you've deleted a file.  And MC will move this file from the Removed database back in your Main database if you ask MC to re-import it.  Or if you removed the entry from the Removed database manually, MC will import this file as if it were new.

That's exactly the behavior it used to have, and seemed to work flawlessly until a few months ago. 

I wouldn't rely on the Recently Imported playlist, since MC pretty much owns this.  Instead, create your own Smartlist that shows you the last N days (or weeks or whatever) of imported files, from most to least recent.

Also, I have posted in the beta forum a view to look at your recently imported files.  I'll post it if it is useful.

Please do.  But if the underlying system doesn't manage/recognize imports correctly, how is any list going to determine the same thing?

--Bill

[MrC]

Right.  But in every case the duplicates refer to the exact same file and path.  In my music hierarchy, which is part of an all-media hierarchy on my library server, there are no duplicates and everything is in E:/FLAC/flacHD/Artist/Album Name/track#-track-title format.  The E: directory is network shared and any other system, including MC Clients, has a local directory E: from the server share and therefore an identical hierarchy.

Can you select two tracks that show this problem and use File > Export Playlist.  Select:

   Format: Text File (CSV delimited)
   Included Fields: All Fields
   Output Range: Selection

Save it to the Desktop, zip it, and attach it in a reply here. 

[bblue]

Can you select two tracks that show this problem and use File > Export Playlist.  Select:

   Format: Text File (CSV delimited)
   Included Fields: All Fields
   Output Range: Selection

Save it to the Desktop, zip it, and attach it in a reply here. 

I can't now, no.  Last night I did a Remove Duplicates on that playlist.  It was one of the Cream Wheels of Fire HD albums, and every field was identical.  When dups show up again I'll forward as per your request.

--Bill

[glynor]

To be clear, you CAN have duplicates in manual Playlists.  This is by design (because you might want to play the same song over and over for some reason).  If you add the same song to a playlist 45 times, it will appear in the playlist 45 times.  That doesn't mean it is in the Library 45 times, it is just "repeated" in that particular playlist over and over.

I actually use this quite a bit, for business purposes.

You SHOULD NOT be able to have the same physical file imported into MC multiple times (eg with different file keys).  There are some (reasonably rare) reports of this occurring recently under certain conditions, that I don't think they've fully quashed.

But, this latter item has absolutely, positively nothing to do with Playlists.  So, referencing both is confusing the issue somewhat.

[bblue]

To be clear, you CAN have duplicates in manual Playlists.  This is by design (because you might want to play the same song over and over for some reason).  If you add the same song to a playlist 45 times, it will appear in the playlist 45 times.  That doesn't mean it is in the Library 45 times, it is just "repeated" in that particular playlist over and over.

I actually use this quite a bit, for business purposes.

You SHOULD NOT be able to have the same physical file imported into MC multiple times (eg with different file keys).  There are some (reasonably rare) reports of this occurring recently under certain conditions, that I don't think they've fully quashed.

But, this latter item has absolutely, positively nothing to do with Playlists.  So, referencing both is confusing the issue somewhat.

Clear as can be.  But the point was that these playlist dups were appearing on their own.  They were NOT being manually added.

--Bill

[MrC]

Bill, we're having a hard time following you.  Specifically, which playlist?

[bblue]

Bill, we're having a hard time following you.  Specifically, which playlist?

I'm only using manual playlists.  They are organized thusly:

ALL Playlist: everything I have in the audio content, except DSD
MAIN Playlist: Non-specific albums/track that aren't in a more specific playlist, for background listening, more or less.
OLDIES Playlist: Only songs from 50's 60's and 70's
Frequent: HD (48k and up, 24-bit and DSD) tracks that are frequently heard
DSD Native: DSF tracks exclusively
Recently Imported: (when it works) a list of tracks which have been imported to or discovered by MC, but not yet assigned to their 'home' playlist

I move tracks around by selecting a group I want to move or add to a different playlist, into tags->playlists and checking or unchecking playlist membership boxes to suit. For example, when there are new adds in Recently Imported I bring up just that playlist and select all items, then add them to the ALL Playlist.  Then select (for example) all new Oldies tracks (still in playlists manager) and check the OLDIES Playlist box.  When I'm done placing things where I want them, I UNcheck the Recently Imported box.  At this point Recently Imported is clear and all tracks are in their appropriate playlists.  It takes very little time to do this, but quickly categorizes them where I want them, and there's no way to duplicate.  Also there's no time limit associated with tracks in Recently Imported.  They stay until I remove them.

I have the Options->General->Advanced->Clear Recently Imported on Exit set to Leave all tracks.   There's another option Don't Import on Play which I can't find at the moment, but it is set to NOT import on Play.

So as far as I can tell, there's no set way in which playlist entries can be duplicated, unless I do it manually and deliberately.  Yet occasionally exact duplicate tracks will appear in my ALL Playlist.

I hope this helps clarify my procedures and setup.  I've been using this method for about two years with completely predictable results until very recently.

--Bill

[MrC]

If I'm understanding this matter correctly, you're saying that MC is adding extra entries into your manually created and managed playlists, and that those duplicates are the same file path (i.e. they are the same physical files).

I've not heard of MC doing this, so my hunch says that you inadvertently added the duplicates.

Looking at what your doing, this seems to me a lot of work, when you can do it much more effectively (and automatically) by using something like Keywords or Genre, or your own field, replacing the manually managed/ammended Playlist with a Smartlist.  You would simply add the Keyword, such as Oldies, or Frequent, and create smartlists that query for these files.  That way, you just tag them directly in any view, and they automatically just appear in the various lists.

[bblue]

If I'm understanding this matter correctly, you're saying that MC is adding extra entries into your manually created and managed playlists, and that those duplicates you have do not have the same file path (i.e. they are different physical files).

I'm not sure what you mean.  The duplicates that appear as if by magic are *identical*.  The same physical files.  I do have a number of duplicate versions of songs that are different files, but those are not an issue.

I've not heard of MC doing this, so my hunch says that you inadvertently added the duplicates.

I guess it could be possible, but I just don't see how.

Looking at what your doing, this seems to me a lot of work, when you can do it much more effectively (and automatically) by using something like Keywords or Genre, or your own field, replacing the manually managed/ammended Playlist with a Smartlist.  You would simply add the Keyword, such as Oldies, or Frequent, and create smartlists that query for these files.  That way, you just tag them directly in any view, and they automatically just appear in the various lists.

This way I don't have to do any keyword editing, and don't depend on what are usually arbitrary assignments in genre and certain other fields.  Plus, I can very easily move them from one playlist to another (or add to another while keeping in the original) without any second guessing about how a smartlist might interpret them.  I depend on things being exactly where I put them.

--Bill

[MrC]

Sorry, I corrected my misstatement above.  I meant Same Physical files.

Since playlists are manually managed, without some verification (like a video) showing that entries just duplicate themselves, I suppose there isn't much to go on here.  It is a frequent occurrence with 'puters that people do things and have no realization or awareness of the act.