INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Thousand duplicates with same path and folder name  (Read 2700 times)

LETRA

  • World Citizen
  • ***
  • Posts: 123
Thousand duplicates with same path and folder name
« on: December 09, 2019, 06:45:52 pm »

Hi. I have a weird problem. There are about two thousand albums in my library duplicated but with the same path and folder name (see picture). Fix broken links does not fix it. Smart list possible duplicates found many, but I can not delete because they are identical names (and thousands). And smart list missing files can't find any. Can you help me solve this?
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72438
  • Where did I put my teeth?
Re: Thousand duplicates with te same patch and folder name
« Reply #1 on: December 09, 2019, 07:44:44 pm »

Hover your mouse over two duplicates.  You should see a difference.
Logged

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #2 on: December 09, 2019, 08:10:21 pm »

Hover your mouse over two duplicates.  You should see a difference.

Do not show more data by moving the mouse near the name of the song. See the picture. The file name and all tags are exactly the same.
Logged

HaWi

  • Citizen of the Universe
  • *****
  • Posts: 946
Re: Thousand duplicates with te same patch and folder name
« Reply #3 on: December 10, 2019, 06:23:23 am »

Could there be special characters in one version of the names that are not visibly different? I have seen something like that in my library.
Logged
rPi5/8GB, Debian 12 Bookworm on SSD | JRMark (33.0.37 64 bit): 2784
MacBookPro (2013), 2.6 GHz Quad-Core Intel Core i7, MacOS 11.7.17 | JRMark (33.0.41 64 bit): 3826
Mac Studio M2 Max, 64GB, 1TB SSD, macOS Sequoia 15.1.1 | JRMark (33.0.41 64 bit): 9056
Docker Container (shiomax) DS1819+ | JRMark (33.0.37 64 bit): 1431
JRemote 3.43
MO 4Media 1.5.7 | Marantz SR7007 (RSL 5.1) HDMI to MacBookPro

blgentry

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 8014
Re: Thousand duplicates with te same patch and folder name
« Reply #4 on: December 10, 2019, 07:57:56 am »

We keep seeing this kind of duplication when CUE files are used.  Do you have CUE files in these album folders?

Brian.
Logged

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #5 on: December 11, 2019, 05:23:34 pm »

Could there be special characters in one version of the names that are not visibly different? I have seen something like that in my library.

It is weirder. How far I have seen are exactly the same. But, for example, click on "locate on disk", on the duplicate entry, blocks jriver and the finder doesn't open.

We keep seeing this kind of duplication when CUE files are used.  Do you have CUE files in these album folders?

Brian.

I don't use cue files. 96% flac, 4% mp3. The problem occurred when I migrated from external disk to a Drobo 5C. I copied the external disks in the Drobo without jriver and then used "rename, copy, move files" to reallocate the library. In the first two thousand albums (of a total of twenty-five thousand) jriver has doubled the entries in the library.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Thousand duplicates with te same patch and folder name
« Reply #6 on: December 11, 2019, 07:23:04 pm »

I don't see any Special Characters in your image, but your issue sounds a lot like the original duplicates problem in this thread: https://yabb.jriver.com/interact/index.php/topic,123158.0.html

It's long and detailed, and may not exactly match your situation. Sorry. But as you copied from a Mac drive to a new drive external to MC, and then did a RM&CF to point MC to the new location, it sounds very similar. "Locate > On Disk (external)" not working is further compelling evidence.

There is a lot of stuff in that thread. Disk format, Code Pages, Character Sets, Compound Characters. Something in there is making the two apparently identical paths different.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #7 on: December 11, 2019, 07:34:19 pm »

I don't see any Special Characters in your image, but your issue sounds a lot like the original duplicates problem in this thread: https://yabb.jriver.com/interact/index.php/topic,123158.0.html

It's long and detailed, and may not exactly match your situation. Sorry. But as you copied from a Mac drive to a new drive external to MC, and then did a RM&CF to point MC to the new location, it sounds very similar. "Locate > On Disk (external)" not working is further compelling evidence.

There is a lot of stuff in that thread. Disk format, Code Pages, Character Sets, Compound Characters. Something in there is making the two apparently identical paths different.

The format for external drives is the same, mac OS Plus. There may be other possibilities, although the problem is in alphabetical order, that is, it affects all the artists of the first letters of the alphabet. I have an idea that is to change the name to Drobo and force JRiver to map all the entries in the library again. It is an easy solution that I can try. Do you think that could work? What would be the correct steps?
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Thousand duplicates with te same patch and folder name
« Reply #8 on: December 11, 2019, 07:50:59 pm »

I helped tangolovers fix her problem by using the RM&CF to rename the path and filenames of the problem files. I don't think just changing the disk name to "Drobo" from ""Drobo 5C" will be enough, if that is what you mean.

Here is a simple test for your problem.

Pick a pair of duplicate files.
Select the one that plays in MC, and works using the "Locate > On Disk (external)" function.
Use the Rename function (F2) to make the [filename] field editable. Check for any gaps or anomalies in the name, as per that thread I linked to. Maybe take an image of what the field looks like when in edit mode.
Select the file that won't play, and doesn't work using the "Locate > On Disk (external)" function.
Use the Rename function (F2) to make the [filename] field editable. Check for any gaps or anomalies in the name, as per that thread I linked to. Maybe take an image of what the field looks like when in edit mode.

Compare the two images or just confirm that the two are still identical when in edit mode.

If you see a difference, that is the problem. The MC Library has an old record that includes either a compound character or not, and MC has imported the file which has the reverse characteristics regarding compound characters.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #9 on: December 11, 2019, 08:06:26 pm »

I helped tangolovers fix her problem by using the RM&CF to rename the path and filenames of the problem files. I don't think just changing the disk name to "Drobo" from ""Drobo 5C" will be enough, if that is what you mean.

Here is a simple test for your problem.

Pick a pair of duplicate files.
Select the one that plays in MC, and works using the "Locate > On Disk (external)" function.
Use the Rename function (F2) to make the [filename] field editable. Check for any gaps or anomalies in the name, as per that thread I linked to. Maybe take an image of what the field looks like when in edit mode.
Select the file that won't play, and doesn't work using the "Locate > On Disk (external)" function.
Use the Rename function (F2) to make the [filename] field editable. Check for any gaps or anomalies in the name, as per that thread I linked to. Maybe take an image of what the field looks like when in edit mode.

Compare the two images or just confirm that the two are still identical when in edit mode.

If you see a difference, that is the problem. The MC Library has an old record that includes either a compound character or not, and MC has imported the file which has the reverse characteristics regarding compound characters.

Tomorrow I follow all the steps and inform you. Thanks for your help.
Logged

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #10 on: December 11, 2019, 08:52:59 pm »

Hello again. I've been seeing what you told me. I am using version 25.0.123. With version 25.0.125 when locating on disk (it may be related or not) JRiver was blocking. Now I can locate the two duplicate entries on the disk, they are identical and point to the same folder, where there is only one version of the files.

When I play the album, each track is played twice. If I mark a rating on a track, it is only visible in one of the two entries (see picture).

I am thinking what the workflow was and I think it was a problem restoring the library backup while the RM&CF process was underway, so I closed the application and used RM&CF again.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Thousand duplicates with te same patch and folder name
« Reply #11 on: December 11, 2019, 09:19:20 pm »

I am thinking what the workflow was and I think it was a problem restoring the library backup while the RM&CF process was underway

Well, that would be very bad.

Given that both entries can be played, and "Locate > On Disk (external)" works for both now, I think the best path forward would be to Restore a Library Backup from before you ran the first RM&CF process, when there were no duplicates but files pointed to the wrong location, and then do the fix using the RM&CF process again. DO BACKUP your Library again before doing the new Restore, so if everything goes wrong you can get back to your current state again.

Your Library is in a strange unknown state at the moment. It sounds like internal file pointers are messed up. So going back to a known state and working forward from there is the best thing to do, I think.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #12 on: December 11, 2019, 09:33:29 pm »

Your Library is in a strange unknown state at the moment. It sounds like internal file pointers are messed up. So going back to a known state and working forward from there is the best thing to do, I think.

The problem is that pre-error backups point to two external disks. I don't have a clean backup with the Drobo library.

And another matter. In these fifteen days I have created about twenty new library views for record labels. And I have worked on the tags and other details, which JRiver sometimes saves in separate files, not in the tags of the flac file itself. Is there a way to export library views, for example?
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Thousand duplicates with te same patch and folder name
« Reply #13 on: December 12, 2019, 02:46:34 am »

I can't advise the best way forward to fix this, but here is some information.

Option 1:
Yes, you can export individual Views in MC Standard View to files and import them into your earlier Library. As long as you haven't created any new fields to use in the Views, they will work straight away. If you have created new fields, you will need to create the new fields in your old Library and then import the Views. The functions are under the "Customise View" right-click menu item, at the bottom of the "Settings" dialogue. The files are saved in "C:\Users\[UserID]\AppData\Roaming\J River\Media Center 25\Library\Saved Views\" for MC25.  Theatre View views are a little more difficult if you built them directly, as there is no equivalent functionality. But if you built a View in MC Standard View, then saved that, imported it into another Library, you can base a Theatre View view on that imported View. Or just take screenshots of any Theatre View only Views, and re-create them in the other Library directly.

If you have done a lot of tagging work that you want to save, if the tags are in the files all you need to do it run the function "Library Tools > Update Library (from tags)" for all the affected files. Of course not all tags are saved into media files or the MC Sidecar files. But you can turn on saving of any tags via the "Options > Library & Folders > Manage Library Fields > {select the tag you want to be saved} > Save in file tags (when possible)" checkbox. For Video and some other files types, that means turning on the Sidecar functionality under "Options > General > Importing & Tagging". There may also be some tags that can't be saved in FLAC files, so use Sidecar file for those files as well. i.e. Enable Audio Sidecar files.

So you could just save all your new or changed Views, save all the tags you want to the FLAC files or Sidecar files if they can't be saved to the FLAC files, go back to your pre-error Library Backup that points to the external disk, update the Library database to point to the new location on the Drobo using RM&CF, and then import the Views and Update Library (from tags) for all affected files.

Option 2:
Or, you could identify which Duplicates are the bad ones, if there is even a difference now that both duplicates play and are found via "Locate > On Disk (external)", and delete one of the duplicates from the Library, but not from the disk.

While the two records in the Library for the duplicates may look identical, they will have different File Keys, I suspect. MC uses File Keys to identify all file in the Library. You can view the integer value of the File Keys by adding an Expression column to a View and using the Expression "FileKey()" without quotes. See https://wiki.jriver.com/index.php/File_Path_and_Identifier_Functions#FileKey.28.E2.80.A6.29 for more information.

If you are seeing an anomaly with the naming of one of the duplicates using the Rename function as I mentioned above, you should be able to decide if the files with low values of File Keys are the bad ones, or the files with high values (more recent files). Based on that, if you can find the range of File Key values that the bad files have, then you can create a View to show only that range, and then delete them from the Library. As always, if trying this method, backup your Library before you start so that you can get back to a known point if needed.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with te same patch and folder name
« Reply #14 on: December 12, 2019, 08:05:18 am »

Wonderful information. Thanks Roderick. I can think of a third option. I hope you can tell me if it is possible.

Duplicates affect all artist albums up to the letter #B or so. Could I delete the entries in the library of all those albums and then get JRiver to rescan the entire disk and add them again without duplicate entries?
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: Thousand duplicates with te same patch and folder name
« Reply #15 on: December 12, 2019, 03:54:19 pm »

Yes, you could do that. It may not be perfect though. If there is metadata in the Library that isn't in file tags, it may get lost. However, it may not. MC keeps a record of all deleted files, and when they are re-imported it will use the metadata from the "Removed Database" it keeps. As long as MC picks the correct duplicate in that process, or both duplicates have the same metadata, then you should be okay.

As always, backup first, and test with a small number of files first. Perhaps take note of any special metadata and confirm it is reinstated during the re-import.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

LETRA

  • World Citizen
  • ***
  • Posts: 123
Re: Thousand duplicates with same path and folder name
« Reply #16 on: December 13, 2019, 03:49:33 pm »

Thanks Roderick. Everything went well. This type of duplicates can be removed from the library in this way and then the correct originals are added again by unchecking the box in / Options / Library & Folders / Import Data / Ignore files previously deleted in the library.

Thank you very much for your help and that of other people in the forum.
Logged
Pages: [1]   Go Up