Topic: Why does import duplicate files? (Read 6085 times)

SeanC · « **on:** October 04, 2004, 01:27:47 pm »

Using v11.0.93, when I import 830 MP3 files, 1650 files are added to the MC library. Most are just duplicated in the library.

I tried clearing the library and re-importing, but I get the same result.

I figure I must be doing something incorrectly - can someone help me out?

RemyJ · « **Reply #1 on:** October 04, 2004, 02:46:22 pm »

This happens to me when I accidentally import a playlist along with a group of files.

Matt · « **Reply #2 on:** October 04, 2004, 02:47:35 pm »

Maybe a silly question, but how do you know it's in the library twice?

Are the filenames identical? Is it possible you're looking at a playlist instead of an unfiltered library? (playlists can have duplicates)

KingSparta · « **Reply #3 on:** October 04, 2004, 03:16:23 pm »

i just re-imported my library (sat) and i did not get dups

using latest mc11

maybe check your settings on the advance tab

IanG · « **Reply #4 on:** October 05, 2004, 12:48:17 pm »

Quote from: Matt on October 04, 2004, 02:47:35 pm

Maybe a silly question, but how do you know it's in the library twice?

Are the filenames identical? Is it possible you're looking at a playlist instead of an unfiltered library? (playlists can have duplicates)

I've got the same problem - the filenames are identical and they're not playlists. It's all the files in some folders. I can't see any pattern to it beyond that.

Ian G.

TAFKAFEX · « **Reply #5 on:** October 05, 2004, 12:57:44 pm »

Quote from: RemyJ on October 04, 2004, 02:46:22 pm

This happens to me when I accidentally import a playlist along with a group of files.

Same here. Doublecheck your folders and eleminate any -- *.m3u --. If you've already imported the folders, delete all existing entries in your database and reimport the folders.

Alex B · « **Reply #6 on:** October 05, 2004, 02:32:02 pm »

Strange, I don't get any duplicate entries when importing files to MC 11.0.93 or MC 10.0.173. I have MP3 files, an M3U playlist and an MPL playlist in the same folder.

Not by right clicking in Windows Explorer, not by Import Media menu or not by MC tree (Drives & Devices). Are there any other ways?

If I import both playlists at the same time I see the same files twice in the Imported Playlists because I imported two playlists, but not in the Media Library.

I can get duplicates only by importing CUE files twice with MC 11 and I think it's a good thing, because those entries can be used as separately tagged virtual links to a single media file.

TAFKAFEX · « **Reply #7 on:** October 05, 2004, 02:50:51 pm »

Drag & drop.

RemyJ · « **Reply #8 on:** October 05, 2004, 04:15:19 pm »

Quote from: Alex B on October 05, 2004, 02:32:02 pm

Funny, I don't get any duplicate entries when importing files to MC 11.0.93 or MC 10.0.173. I have MP3 files, an M3U playlist and an MPL playlist in the same folder.

Not by right clicking in Windows Explorer, not by Import Media menu or not by MC tree (Drives & Devices). Are there any other ways?

If I import both playlists at the same time I see the same files twice in the Imported Playlists because I imported two playlists, but not in the Media Library.

I can get duplicates only by importing CUE files twice with MC 11 and I think it's a good thing, because those entries can be used as separately tagged virtual links to a single media file.

Example:

If you use MC to create a data CD, it also writes a playlist to the CD. If you take that CD and copy the contents to a hard drive, the filenames in the playlist will no longer agree with the actual filenames. If you then import the directory with MC, the music files get imported, the playlist gets imported, then placeholders get created for each track in the playlist that MC couldn't find. The result is duplicate track entries.

This isn't something that needs to be fixed, just something to watch out for.

Alex B · « **Reply #9 on:** October 05, 2004, 04:59:07 pm »

Quote from: RemyJ on October 05, 2004, 04:15:19 pm

If you take that CD and copy the contents to a hard drive, the filenames in the playlist will no longer agree with the actual filenames. If you then import the directory with MC, the music files get imported, the playlist gets imported, then placeholders get created for each track in the playlist that MC couldn't find. The result is duplicate track entries.

I see. Technically they are not duplicate entries because the most important library field, "Filename" is different. MC uses "Filename" fields for separating entries. The filenames (& possible paths) are correct in my playlists.

Quote from: TafkaFex on October 05, 2004, 02:50:51 pm

Drag & drop.

Yes, D & D too.

I don't get duplicates with MC 11, but there is something strange with playlist importing. Sometimes importing an M3U playlist imports music files and makes a new entry to the Imported Playlists (as it should). Sometimes the same M3U imports as a new document (like .txt and others) and becomes an entry in the main library. I have cleared the library between the import sessions. "Exclude previously removed" import option is off.

MC 11 also keeps files reserved after removing from the library. I have noticed many times that I have to close MC before I can delete or edit files with other software.

KeystoneCop · « **Reply #10 on:** October 05, 2004, 05:21:43 pm »

are you sure it's from import?

I have been moving a lot of files around.

Just did a find duplicates on "filename" had over 9000 so i don't think it has anything to do with errors.

I deleted them and imported again.

Alex B · « **Reply #11 on:** October 05, 2004, 06:00:07 pm »

This is the contents of a "Filename" field in MC library:

"D:\Music\Tina Arena\Just Me\Tina Arena - Just Me - 10 - Something's Gotta Change.ape"

You cannot have that field duplicated in MC library unless the library is broken somehow.

KeystoneCop · « **Reply #12 on:** October 05, 2004, 06:02:07 pm »

I agree something is broken

If it helps.. I move things from drive to drive.

Alex B · « **Reply #13 on:** October 05, 2004, 06:49:43 pm »

Quote

I move things from drive to drive.

Then at least the drive letter part should be different. (The "Filename" field information is different if there are any changes and then the entries are not duplicate.)

So, if you had e.g. just one duplicate of each file you got over 4500 files which were duplicated in your library. Or did you have 9000 copies of one file?

I make library backups all the time. Usually before and after major changes. I keep about ten recent backup versions in a safe place, so I can always go back if the library gets corrupted. Though that has never happened.

SeanC · « **Reply #14 on:** October 05, 2004, 09:38:26 pm »

Quote from: RemyJ on October 05, 2004, 04:15:19 pm

If you use MC to create a data CD, it also writes a playlist to the CD. If you take that CD and copy the contents to a hard drive, the filenames in the playlist will no longer agree with the actual filenames. If you then import the directory with MC, the music files get imported, the playlist gets imported, then placeholders get created for each track in the playlist that MC couldn't find. The result is duplicate track entries.

This was exactly my circumstance. I still don't see why having a playlist would cause the music files to show up in the library twice, but I'm happy to delete the m3u and have the problem solved.

IanG · « **Reply #15 on:** October 06, 2004, 04:31:37 pm »

It looks like my problem's with the library itself. I loaded a different library and (re)imported all my music - some updates, but no duplicates. I deleted the duplicates from the original library, did an import & voila - the same duplicates are back!

Ian G.

KeystoneCop · « **Reply #16 on:** October 06, 2004, 05:54:37 pm »

I have tried to recreate the problem. With no success (it would be an easy fix if it was repeatable. (I could do it and Matt would be out of a job)).

I had about 4500 duplicate files. This was my first install of version 11 and I said share the library with 10 (after a few hours use I don't see why, I am ready to pay for version 11).

I moved a lot of files using rename files from properties. I also moved some folders using windows XP. I have had this problem with a few files in the past when they have been in use.

It appears Media center makes changes to the library even if an error is encountered during execution.

Maybe a Timing Problem... Just guessing

KeystoneCop · « **Reply #17 on:** October 06, 2004, 06:15:47 pm »

It has to be a timing thing. I moved a folder, checked for duplicate file . NONE

now I checked and found 2 duplicate files.

IanG · « **Reply #18 on:** October 07, 2004, 02:44:29 am »

I tried deleting the duplicates, backing up my library, deleting it and restoring to a new one. No luck - I have exactly the same problem with the new library!

Ian G.

KeystoneCop · « **Reply #19 on:** October 07, 2004, 09:36:29 am »

OK. This is crazy.

Turned on the computer and updated the library.
Now I have 42 duplictaes (21 files). all the files that I copied yesterday are now showing up as duplicates.

They were not there after a library update yesterday. But latter in the day 4 duplicates (2 files showed up).

Is there anyway to look at a key for these records?

Alex B · « **Reply #20 on:** October 07, 2004, 10:13:04 am »

I suppose you have the duplicates in the MC library. You don't have duplicates on your hard drive (duplicate copies of the music files with a bit different filename or in a different location). Correct?

Could you make a playlist of one track and the duplicate? Export it to MPL format, open with Notepad and paste here, like this:

Quote

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0">
<Item>
<Field Name="Filename">E:\Music\The Rasmus\The Rasmus - Ghostbusters.mp3</Field>
<Field Name="Artist">The Rasmus</Field>
<Field Name="Album">Hellofacollection</Field>
<Field Name="Name">Ghostbusters</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Rock</Field>
<Field Name="Date">36892,0000000000000000</Field>
<Field Name="Comment">090ED112</Field>
<Field Name="Bitrate">237</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="File Size">6509797</Field>
<Field Name="Duration">214</Field>
<Field Name="Track #">12</Field>
<Field Name="Date Created">1077307477</Field>
<Field Name="Date Modified">1023386930</Field>
<Field Name="Date Imported">1082391712</Field>
</Item>
<Item>
<Field Name="Filename">E:\Music\The Rasmus\The Rasmus - Ghostbusters.mp3</Field>
<Field Name="Artist">The Rasmus</Field>
<Field Name="Album">Hellofacollection</Field>
<Field Name="Name">Ghostbusters</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Rock</Field>
<Field Name="Date">36892,0000000000000000</Field>
<Field Name="Comment">090ED112</Field>
<Field Name="Bitrate">237</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="File Size">6509797</Field>
<Field Name="Duration">214</Field>
<Field Name="Track #">12</Field>
<Field Name="Date Created">1077307477</Field>
<Field Name="Date Modified">1023386930</Field>
<Field Name="Date Imported">1082391712</Field>
</Item>

KeystoneCop · « **Reply #21 on:** October 07, 2004, 11:20:49 am »

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0">
<Item>
<Field Name="Filename">F:\Bible\Ot22-Song Of Solomon\01-Song Of Songs.mp3</Field>
<Field Name="Artist">Max Mclean</Field>
<Field Name="Album">Ot22-Song Of Solomon</Field>
<Field Name="Name">Song Of Songs</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Vocal</Field>
<Field Name="Date">37622</Field>
<Field Name="Bitrate">96</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="File Size">1707993</Field>
<Field Name="Duration">142</Field>
<Field Name="Track #">1</Field>
<Field Name="Date Created">1096559775</Field>
<Field Name="Date Modified">1096719348</Field>
<Field Name="Date Imported">1097161393</Field>
</Item>
<Item>
<Field Name="Filename">F:\Bible\Ot22-Song Of Solomon\01-Song Of Songs.mp3</Field>
<Field Name="Artist">Max Mclean</Field>
<Field Name="Album">Ot22-Song Of Solomon</Field>
<Field Name="Name">Song Of Songs</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Vocal</Field>
<Field Name="Date">37622</Field>
<Field Name="Bitrate">96</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="Last Played">1096566306</Field>
<Field Name="File Size">1707993</Field>
<Field Name="Duration">142</Field>
<Field Name="Number Plays">1</Field>
<Field Name="Track #">1</Field>
<Field Name="Date Created">1096559775</Field>
<Field Name="Date Modified">1096719348</Field>
<Field Name="Date Imported">1096561633</Field>
<Field Name="MD5TextHash">D6128F6C7DF44BC5C2E8FD7D9630B159</Field>
</Item>
</MPL>

Maybe this is a clue only one of the files showed the MD5TEXTHASH field

It is the same for all the others. and the duplicate count keeps growing.

Alex B · « **Reply #22 on:** October 07, 2004, 11:30:15 am »

What files do you have in the "F:\Bible\Ot22-Song Of Solomon" folder?

KeystoneCop · « **Reply #23 on:** October 07, 2004, 11:34:13 am »

This is happening in all kinds of folders. I now think it has something to do with custom fields. only files in folder are mp3.

IanG · « **Reply #24 on:** October 07, 2004, 11:44:02 am »

Quote from: Alex B on October 07, 2004, 10:13:04 am

Could you make a playlist of one track and the duplicate? Export it to MPL format, open with Notepad and paste here, like this:

This is insane! Last night I had a rock solid error - today I go into MC to follow these instructions, and no duplicates!!!

Ian G.

Alex B · « **Reply #25 on:** October 07, 2004, 11:48:22 am »

Quote from: KeystoneCop on October 07, 2004, 11:20:49 am

<Field Name="MD5TextHash">D6128F6C7DF44BC5C2E8FD7D9630B159</Field>

The field comes from King Sparta's MD5 Master plugin, I think. Maybe the plugin has something to do with this. I have not used it and I don't know how it works.

KeystoneCop · « **Reply #26 on:** October 07, 2004, 01:32:19 pm »

I renamed a series of files. (none of which were on the duplicate list)

Then updated the library, and the list of duplicate files changed... 40 or so that were listed are no longer listed, but it added some others.

I just keep trying to describe what’s happening in hopes it will help someone fix it.

Anone from JRiver care to tell me if this helps. ?

KeystoneCop · « **Reply #27 on:** October 07, 2004, 01:40:41 pm »

Next I just did a clean file properties.. album name track (changed 140 files)

after it said it was completed. hade 40 more duplicate files listed.

IanG · « **Reply #28 on:** October 07, 2004, 01:54:00 pm »

Quote from: KeystoneCop on October 07, 2004, 01:32:19 pm

I renamed a series of files. (none of which were on the duplicate list)

At last - a coincidence! My problems started just after I'd renamed a load of files from the properties.

Ian G.

Matt · « **Reply #29 on:** October 07, 2004, 02:23:24 pm »

We're unable to duplicate this.

Is it possible MC is still working on the rename when you're updating the library? (look at the statusbar to see)

KeystoneCop · « **Reply #30 on:** October 07, 2004, 02:35:08 pm »

Thanks for the reply Matt.

I have waited for the status to complete.

I am now finding different files as duplicates even when I don't modify any.

The only thing I see in common is one file always has the md5texthash field.

My first version 11 was 93. I said to use the old library. I can delete the whole library and start over without and cutom made feels if you think that might help.

KingSparta · « **Reply #31 on:** October 07, 2004, 02:41:37 pm »

Quote from: Alex B on October 07, 2004, 11:48:22 am

Quote from: KeystoneCop on October 07, 2004, 11:20:49 am
<Field Name="MD5TextHash">D6128F6C7DF44BC5C2E8FD7D9630B159</Field>

The field comes from King Sparta's MD5 Master plugin, I think. Maybe the plugin has something to do with this. I have not used it and I don't know how it works.

that field is for MD5 Master

the plug-in however does not make Duplicate Files

and i hope he is not basing he has duplicates and this thread on this, if so he would be in error on this whole subject.

KeystoneCop · « **Reply #32 on:** October 07, 2004, 02:44:24 pm »

more info the last modified date on all the files shown as duplicates are at least one day prior to my installing version 93 (my first try at version 11)

but i did move the files that show up as duplicates with windows, and had MC update the library.

KeystoneCop · « **Reply #33 on:** October 07, 2004, 02:46:54 pm »

King.. I don't think MD5 is the problem.. but i had the entire file with that field in populated. all the duplicate files never showed up in version 10. only version 11 and the duplicate file does not contain the md5 fields.

KingSparta · « **Reply #34 on:** October 07, 2004, 02:51:14 pm »

Quote

King.. I don't think MD5 is the problem..

I just don't Care for comments that has no bases in fact, thats the only reason i said anything.

now back to your program that is already in progress

KeystoneCop · « **Reply #35 on:** October 07, 2004, 03:05:36 pm »

field (filename).jmd I checked this file and there is only one entry for the files that show up as duplicated. and i do see I added some md5text after running with version 11.

The Library is 54 megs so too big for emailing to you.

Matt · « **Reply #36 on:** October 07, 2004, 03:05:38 pm »

The problem isn't extra fields or King's plugin.

Remove all duplicates. Then search for dupes again adding ~d=a to the search, and remove again. (this step is important)

Then give a step-by-step to reproduce if you're still getting duplicates.

KeystoneCop · « **Reply #37 on:** October 07, 2004, 03:17:50 pm »

Thanks.

I got 11,000 plus duplicates. I am going to just delete the entire library and start over. maybe when i first started i was to quick on the trigger and got everything confussed.

And King is the Greatest...We all know that....

KeystoneCop · « **Reply #38 on:** October 07, 2004, 04:08:18 pm »

mc library folder now 17 megs no duplicates. sorry for all the trouble. but i still don't have a clue what went wrong unless i went to fast. i usally do.

KeystoneCop · « **Reply #39 on:** October 07, 2004, 04:20:08 pm »

at this point i am going to do a I AM SORRY.

Sorry that everyone has read all this stuff and I didn't do a better job of testing.

latest is (I cannot say all) but most of the files that I changed while on version 11 have lost the name field

(I deleted the library in windows because when i said remove the library file , it stayed the same size.

Don't take me wrong beta is beta and I don't have a problem fixing things up.

IanG · « **Reply #40 on:** October 07, 2004, 04:23:13 pm »

I did another import and they're back! Here's an MPL playlist.
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<MPL Version="2.0">
<Item>
<Field Name="Filename">D:\Media\Music\Gordon Giltrap\Collection - Disk 2\Here Comes The Sun.mp3</Field>
<Field Name="Artist">Gordon Giltrap</Field>
<Field Name="Album">Collection Disk 2</Field>
<Field Name="Name">Here Comes The Sun</Field>
<Field Name="Band">Collection Disk 2</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Folk</Field>
<Field Name="Date">37987</Field>
<Field Name="Bitrate">99</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="File Size">266368</Field>
<Field Name="Duration">21</Field>
<Field Name="Track #">13</Field>
<Field Name="Date Created">1090739430</Field>
<Field Name="Date Modified">1096397456</Field>
<Field Name="Date Imported">1097183799</Field>
</Item>
<Item>
<Field Name="Filename">D:\Media\Music\Gordon Giltrap\Collection - Disk 2\Here Comes The Sun.mp3</Field>
<Field Name="Artist">Gordon Giltrap</Field>
<Field Name="Album">Collection Disk 2</Field>
<Field Name="Name">Here Comes The Sun</Field>
<Field Name="Band">Collection Disk 2</Field>
<Field Name="File Type">mp3</Field>
<Field Name="Genre">Folk</Field>
<Field Name="Date">37987</Field>
<Field Name="Bitrate">99</Field>
<Field Name="Media Type">Audio</Field>
<Field Name="Last Played">1090880670</Field>
<Field Name="File Size">266368</Field>
<Field Name="Duration">21</Field>
<Field Name="Number Plays">4</Field>
<Field Name="Track #">13</Field>
<Field Name="Date Created">1090739430</Field>
<Field Name="Date Modified">1096397456</Field>
<Field Name="Date Imported">1079735772</Field>
</Item>
</MPL>

And now I'll try Matt's suggestion.

Ian G.

IanG · « **Reply #41 on:** October 07, 2004, 05:34:27 pm »

Well, that seems to have done the trick - I don't know what ~d=a does, but it flushed out all sorts of rubbish, including the old filenames (pre-rename) and lots of CD entries.

Ian G.

IanG · « **Reply #42 on:** October 08, 2004, 10:02:11 am »

So what does "~d=a" do? Is it documented anywhere?

Ian G

INTERACT FORUM

Author Topic: Why does import duplicate files? (Read 6085 times)

TAFKAFEX

TAFKAFEX