INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Removing duplicates?  (Read 1792 times)

GreggP

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 321
Removing duplicates?
« on: December 11, 2021, 11:45:44 am »

I recently moved my media to a new drive (actually, I retired my 20+ yr old unRAID media server with an external drive directly attached to my PC and added cloud backup). I used the "Rename, Move, and Copy Files..." feature to point my database to a new location. That worked fine.

I also changed the 'File Location' in Options to the new location. And I configured Auto-Import to monitor the new location.

So now I have duplicate records of a lot of my audio tracks. The initial "Rename, Move, and Copy Files..." worked but I ended up with duplicate entries, possibly from the auto-import setting. It must not have recognized some of the media as already there. The original tracks have a few metadata fields that I added to help me create a few smartlists. I've added custom metadata fields for 'AMG Rating' and 'Date Recorded'. I provide this data whenever I rip/add media to my library. This way, a compilation that was released in 2019 but has a track that was originally recorded in 1968, will be included in a smartlist of music from the 60s. Without this addition, all of these tracks would use the date automatically gathered by the online databases, which is usually wrong for compilations.

When I compare the original to the duplicate I can see that the duplicate doesn't have data entered for my custom fields.

Is there a way I can easily delete these duplicates from my library? There are over 30,000 of them and it would be very tedious to do this manually.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72444
  • Where did I put my teeth?
Re: Removing duplicates?
« Reply #1 on: December 11, 2021, 01:25:06 pm »

Compare the full paths of two duplicates.  Are they different?
Logged

blgentry

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 8014
Re: Removing duplicates?
« Reply #2 on: December 11, 2021, 03:08:03 pm »

I would restore MC's database from a backup and start again with your RM&C.  MC makes automatic backups pretty frequently.  You probably have a dozen of them or more sitting on disk.

Brian.
Logged

GreggP

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 321
Re: Removing duplicates?
« Reply #3 on: December 12, 2021, 12:37:06 pm »

Compare the full paths of two duplicates.  Are they different?

No. The paths are identical. That's the first thing I checked. Probably should've mentioned that but thought it obvious.
Logged

GreggP

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 321
Re: Removing duplicates?
« Reply #4 on: December 12, 2021, 12:47:22 pm »

I would restore MC's database from a backup and start again with your RM&C.  MC makes automatic backups pretty frequently.  You probably have a dozen of them or more sitting on disk.

Brian.

I'll give that a try. I moved my files on 11/29 and I have a backup from 11/28, so I'll use that. Then I'll try the RM&C again. It will be interesting to see if it auto-imports the duplicates again. I will report back if it does.

update1: I just finished restoring the library from 11/28. This time when I ran the 'Rename, Move, & Copy Files' utility, I unchecked the boxes for options except 'Find & Replace' and entered "\\Media\Music\" for 'Find What:' and "F:\Music\" for 'Replace With:'. I think I left the default options checked during my previous RM&C attempt. I noticed that prior to this restore, there were over 90,000 audio tracks. After completing the restore today, there are 60,762 audio tracks. So I was right, there were around 30,000 duplicates. Again, we'll see if MC generates some duplicate entries during auto-import.

Should I turn off auto-import? I hesitate to do this because I know there are about a dozen albums I imported since the 11/29 move and would like it if they were imported automatically.

update2: Auto-import didn't run automatically after restoring the database, so I manually activated it, and over 100 new tracks were found (which is about right). It looks like there are no duplicates this time.
Logged

blgentry

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 8014
Re: Removing duplicates?
« Reply #5 on: December 12, 2021, 07:49:59 pm »

Awesome!

Brian.
Logged

imeric

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1476
Re: Removing duplicates?
« Reply #6 on: January 14, 2022, 02:06:29 pm »

I have a similar issue where MC will keep reimporting the same file(s) twice.

I sometimes re-rip CDs and if the name is slightly different I use RM&C (Update Location) or manually change the path of the file in the tag then Update Library from Tag.  I do this to keep play count and date imported intact...

I also do this to replace lossy versions with lossless...This part works well.

However sometimes the same files will reimport and when deleting the "newly imported" (and already existing) file from library it will keep reimporting itself eventhough there's an existing version of the file in the library.

After 2-3 times of deleting from Library it will eventually work but sometimes the only way to resolve this is to "Remove from Media Library and send to recycle bin", then do a restore from the bin...

Is there a way to create a smartlist with dupes pointing to the same file? I bet I have a few of those in my library...
Logged

imeric

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1476
Re: Removing duplicates?
« Reply #7 on: January 20, 2022, 08:10:58 am »

This just happened again see pic attached.

Using the following smartlist for exact dupes: 
Code: [Select]
[Media Type]=[Audio] ~dup=[Filename]
Logged

imeric

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1476
Re: Removing duplicates?
« Reply #8 on: January 20, 2022, 09:26:34 am »

I then "Remove from library" and the files reappear after re-running auto-import:

Code: [Select]
Library now has 91345 files. Search and update took 0:02.

Imported 10 new files.
Updated 10 files that had external changes.

In-depth details:
Imported:
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\01. Van Halen - The Seventh Seal.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\02. Van Halen - Can't Stop Lovin' You.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\03. Van Halen - Don't Tell Me (What Love Can Do).m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\06. Van Halen - Strung Out.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\05. Van Halen - Big Fat Money.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\04. Van Halen - Amsterdam.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\09. Van Halen - Doin' Time.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\08. Van Halen - Aftershock.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\07. Van Halen - Not Enough.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\10. Van Halen - Baluchitherium.m4a


Updated:
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\01. Van Halen - The Seventh Seal.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\02. Van Halen - Can't Stop Lovin' You.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\03. Van Halen - Don't Tell Me (What Love Can Do).m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\04. Van Halen - Amsterdam.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\05. Van Halen - Big Fat Money.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\06. Van Halen - Strung Out.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\07. Van Halen - Not Enough.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\08. Van Halen - Aftershock.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\09. Van Halen - Doin' Time.m4a
    E:\iTunes Media\ALAC\Van Halen\1995 - Balance\10. Van Halen - Baluchitherium.m4a



And this will keep happening unless I send the files to the recycling bin and restore..
I sent a log to support at jriver on this sequence thx!
Logged
Pages: [1]   Go Up