Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Gatherum]

Adapted from my other thread:

I found it. It's working beautifully.

...Except, one problem: how is it that [Disc #] is still not respected during handheld syncs?

Example: I have the limited edition of Antimatter's album, Fear of a Unique Identity, which has two discs. Track 4 on both discs is a version of "Firewalking". I have a separate field that I use for specifying track version, so that information does not appear in [Name]. This results in track 4 on disc 2 being treated as a duplicate.

Can this be fixed? This has been an issue literally for years.

Pretty please?

[ferday]

while i haven't really had a personal issue with this so far, i agree that disc # is an important part in determining duplicates

so +1

[blgentry]

I don't personally see any good reason for the Handheld Sync system to try to detect duplicates.  Why do it at all?  Just turn that part of the code off.  Unless there's some compelling reason I'm unaware of?

Brian.

[Gatherum]

That's actually a fair point: Why should it not be the user's responsibility to ensure that there are no duplicates?
At the very least, I'd include this as an option that can be turned on or off by the user as needed.

[ferday]

i should clarify my +1

i think the [disc #] should be involved in duplicate detection overall, pursuant to the thread on the mac board

even better would be to allow us to make the dupe expression as we see fit

[blgentry]

It's just very strange that duplicate detection is used at all.  As far as I know there's no other place in MC where duplicates are detected.  People ask for it pretty often as a maintenance tool and they are always pointed to a stock smartlist, or told to make their own.

I just wonder why this feature is included in Handheld Sync.  Puzzling to me.

Brian.

[ferday]

Maybe I don't understand the concept, but dupe detection seems like a mandatory feature when syncing a handheld (iPod in my case) so as not to get the same tracks over and over when doing a mass sync

[blgentry]

Maybe I don't understand the concept, but dupe detection seems like a mandatory feature when syncing a handheld (iPod in my case) so as not to get the same tracks over and over when doing a mass sync

I don't understand.  In what scenario would you set up Handheld sync in such a way to get lots of dupes?  Certainly not from syncing several playlists.  What situation would produce a handheld sync that you would want, but would give you lots of dupes?  If you have dupes, you clearly want them in your library for some reason right?

Brian.

[ferday]

Blgentry i'm not sure we're talking about the same thing (my bad if so!)

when i sync my ipod, there's stuff on it already.  when i want to change the sync i don't want the same tracks to sync on again that are already on there, so i want dupe removal.

but i also have a bunch of remasters etc. that will have the same track names but some tag will be different so i'd like to sync those

[Gatherum]

That's what handheld sync definitions are for. See here.

However handheld sync's duplication prevention still gets in the way, as we've gone over.

[ferday]

re reading that thread, i think blgentry is right on this.  dupe prevention isn't needed in the handheld area.  i don't sync often, so i haven't gotten into the definitions but i'm looking into it now.  as i said i've not personally had any issues yet

[Gatherum]

No thoughts from JRiver staff?

[Gatherum]

Hello?

I'm not letting this one go without some kind of response. Not to be an not a very nice person or anything, but this has been suggested by a variety of users over a vast period, now. I think it deserves at least an affirmation that it cannot be done.

[Gatherum]

Hello, once again?

[RoderickGI]

Carrying on from over here: http://yabb.jriver.com/interact/index.php?topic=105314.msg734097#msg734097

My file naming convention solves the problem you are having here, but have you tried setting the Audio Path in your Handheld options to something like [Album Artist (auto)]\[Album]\[Disc #]\ and the files that are named the same, but are not duplicates and are on different discs, would be placed into different subdirectories on the Handheld Device.

Of course you would probably have to modify the above Audio Path to take into account the case where there is no disc number, as I don't think the sync would handle a directory that includes '\\' very well. It might but I haven't tested that. So a conditional inclusion of the [Disc #] field would be required. An audio path something like;

[Album Artist (auto)]\[Album]\IF(isempty([Disc #], "", 1), , [Disc #]\)

might work. Certainly I can enter the above into the Audio Path field. Unfortunately I don't have the data in my test library to see if it works just at the moment. I tested that in an Expression Column in a view and it worked fine. Actually I was expecting to need to escape the \, using the form [Album Artist (auto)]\[Album]\IF(isempty([Disc #], "", 1), , [Disc #]\\)but the escape wasn't needed in the Expression Column. It may be needed in the audio path definition for the Handheld.

Also, while Playlists are synchronised to the Handheld Device, I don't know if they are modified for changes to the path to the audio files, through the inclusion of [Disc #] in the path.

So a couple of things to test.

[blgentry]

Carrying on from over here: http://yabb.jriver.com/interact/index.php?topic=105314.msg734097#msg734097

My file naming convention solves the problem you are having here, but have you tried setting the Audio Path in your Handheld options to something like [Album Artist (auto)]\[Album]\[Disc #]\ and the files that are named the same, but are not duplicates and are on different discs, would be placed into different subdirectories on the Handheld Device.

You're misunderstanding the problem, which is understandable because you haven't found the test cases in your library to trigger it.  This bug/behavior has nothing to do with file naming.  It's actually a duplicate detection problem.  If MC sees [Name], [Album], and [Track] the same on two songs, it marks them as duplicates and only copies one of them to the handheld.  This is an unusual trigger I'll admit.  I found it with a live album that has 4 discs.  On 2 of the discs, track #1 is called "intro".  MC calls these duplicates when they clearly are not:  They are on different discs.  This is the bug/behavior that we would like to see eliminated.

Brian.

[DJLegba]

^ It's not at all unusual for audiobooks, where you'll have multiple discs with the tracks named "Track 1", "Track 2", etc.

[Gatherum]

have you tried setting the Audio Path in your Handheld options to something like [Album Artist (auto)]\[Album]\[Disc #]\ and the files that are named the same, but are not duplicates and are on different discs, would be placed into different subdirectories on the Handheld Device.

I do, actually; however, MC's duplicate detection algorithm/parameter/whatever doesn't care about output filename or directory path. Instead, it takes into account the library metadata. Because it doesn't respect [Disc #], it doesn't detect that two given tracks with with same artist, album, and track #, yet different disc #'s, are not duplicates of each other.

[RoderickGI]

If MC sees [Name], [Album], and [Track] the same on two songs, it marks them as duplicates and only copies one of them to the handheld.

Ah yes, I see where I was misunderstanding. Duplicate Detection, both generally and for Handheld syncs, doesn't care about the file name.

I'm still cleaning up a huge music collection, with lots of duplication, and haven't really hit this problem. I can think of arguments both why Disc # should be used for de-duplication, and why it shouldn't, depending on the situation. Which is why I merely use MC to show me potential duplicates, and make the final decision myself. I don't have any duplicates used in my Handheld sync though.

Carry on then. Don't mind me. I learned something, so all it good.

[blgentry]

I'm still cleaning up a huge music collection, with lots of duplication, and haven't really hit this problem.
[...]
Which is why I merely use MC to show me potential duplicates, and make the final decision myself.

Have you tried my Duplicated Finder View?  I ask only partially out of vanity.  I mostly ask because I haven't gotten much feedback.  I'm particularly interested in how it works on "huge collections with lots of duplicates".

Brian

[RoderickGI]

I did see that thread the other day, and thought I could try it in the future.

The problem is that much of my collection was inherited from my brother, and it is a mess. Not a lot of metadata, and not very accurate. Lots of partial albums, only including tracks he liked. Plus he seems to have backed up the same song in multiple places on the same drive. Inconsistent file naming. A mess.

So I am gradually working through the drives I have with MusicBrainz Picard to add enough metadata to actually use such a tool. Right now your view wouldn't have enough metadata to work on. Not well anyway.

I guess I should install it just to see what it looks like with my data. It might help with some of it. I have saved the thread URL so I can find it again.

[Gatherum]

Just tried to sync disc four of Anathema's compilation, Fine Days: 1999-2004, the fourth disc of which is Were You There?, a live album. The first two tracks won't sync because they match the first two tracks on disc 3, which is the studio album, A Natural Disaster.

Can this be fixed? I think we have more than enough in evidence and reasoning to justify this being addressed, or at least responded to.

[RoderickGI]

If MC sees [Name], [Album], and [Track] the same on two songs, it marks them as duplicates and only copies one of them to the handheld.  This is an unusual trigger I'll admit.

Another minor clarification if I may.

I just did a little test with a three disc album, and changed the [Name] of the first track of each to match the first disc, so [Track #] 1 was called "Intro" on each. Also, of course, the [Album] was the same for each. So my test met the criteria of matching [Name], [Album], and [Track #]. In fact, [Album Artist (auto)] and [Album Artist] were also the same for all three tracks, and were both (Multiple Artists). The [Disc #] fields were 1, 2, and 3 respectively.

All three synchronised across properly. I was synchronising to a folder on my hard drive if it makes any difference. It was good to see that my Audio Path equation, [Album Artist (auto)]\[Album]\IF(isempty([Disc #], "", 1), , [Disc #]\)\ also worked as expected. Also the Playlist created on the Handheld respected the changed directory structure that now included a Disc number. i.e. "F:\iPhone 3GS\(Multiple Artists)\21st Century Breaks\1", or in the Playlist;
.\..\(Multiple Artists)\21st Century Breaks\1\1-01 Soul of Man - Intro (1).mp3
.\..\(Multiple Artists)\21st Century Breaks\2\2-01 C83 - Back in the Day.mp3
.\..\(Multiple Artists)\21st Century Breaks\3\3-01 AMB - Romeo.mp3

Note that the filename is different for each, since that isn't supposed to have any effect.

But then I realised that the [Artist] field was different for each track. So I made the [Artist] field the same, and re-ran the synchronisation. This time only the first "Intro" track synchronised. I tried changing the [Album Artist (auto)] and [Album Artist] for each track, but that didn't force the duplicate to sync across.

Conclusion: The [Artist] tag is also used by the Handheld synchronisation de-duplication process. So at least the [Name], [Album], [Artist], and [Track #] are used.


I think I agree with you guys that it should be possible to define your own de-duplication expression for Handheld syncs, in the Handheld Options, with a default for users who don't want anything different to how it works now.

[RoderickGI]

BTW, reading back through the thread, I could easily see a user synchronising five Playlists or Smartlists to a Handheld device, and have the same Album on two or more Play/Smartlists.

Hence, there is a need for de-duplication in synchronising to a Handheld.

[Gatherum]

BTW, reading back through the thread, I could easily see a user synchronising five Playlists or Smartlists to a Handheld device, and have the same Album on two or more Play/Smartlists.

Hence, there is a need for de-duplication in synchronising to a Handheld.

Sure. But the following should also be true:

1. The user should be able to turn it off at their leisure.
2. It should respect [Disc #].

I've been considering this issue for the past few days, and I agree that duplication prevention should not completely disappear, but it needs tweaks and options. As it stands, it is simply not at all user-friendly.

[RoderickGI]

Agreed. I would like to use the Handheld functionality more in the future, so I would like this aspect improved.

A Checkbox to turn it on.
When on, a editable field opens with the default de-duplication expression in place.

[blgentry]

BTW, reading back through the thread, I could easily see a user synchronising five Playlists or Smartlists to a Handheld device, and have the same Album on two or more Play/Smartlists.

Hence, there is a need for de-duplication in synchronising to a Handheld.

No, that doesn't make sense.  Why?  Because Handheld Sync uses a common area for writing all songs.  I.E., If I have the same song in two playlists, MC only puts the song in one place.  It doesn't try to write two copies of the song to two different places.  It follows the rules set up in Files paths and more > Audio Path .  This defaults to Artist\Album .

The worst that would happen is that MC's logic would make it try to copy the same file to the same location twice.  In which case, MC should deal with file level duplicates.  Maybe that's the issue:  Maybe Handheld Sync's file level duplicates handling only creates duplicates.

I've actually got a handheld definition right now which duplicates files every single time I run it.  I wonder if this behavior I'm seeing (file level duplicates) is related to this decision to do Handheld Sync de-duping.  In my opinion it would be smarter to have file level duplicate code that has the option to overwrite or ignore.

Brian.

[RoderickGI]

I don't understand.  In what scenario would you set up Handheld sync in such a way to get lots of dupes?  Certainly not from syncing several playlists.  What situation would produce a handheld sync that you would want, but would give you lots of dupes?

Mainly I was responding to this comment Brian. You asked how you could get dups in a Handheld sync. I explained how it could happen simply, and probably quite commonly. Not everyone curates their music collection carefully, or their Playlists. Of course then as you say below, you have a handheld definition that produces dups.

No, that doesn't make sense.  Why?  Because Handheld Sync uses a common area for writing all songs.

Yes I thought that through as well, and you are correct. But if (proper) de-duping reduced the amount of files to be copied over, at the simple level that would reduce the time it takes to synchronise, which I for one would appreciate.

The worst that would happen is that MC's logic would make it try to copy the same file to the same location twice.  In which case, MC should deal with file level duplicates.  Maybe that's the issue:  Maybe Handheld Sync's file level duplicates handling only creates duplicates.

Unfortunately JRiver doesn't have any control over how a handheld device responds when something tries to copy the same file to it twice, into the same location. They have to support as many devices as they can, and can't know how each will respond.

In the best case, which is when the handheld is treated by the PC as a hard drive, Windows will rename the second file by adding " (1)" to it. I saw that yesterday, but I was writing to a hard disk folder. If I was writing to a handheld that presented as a folder I assume it would do the same, but I can't test that.

In the worst case (possibly), the handheld would flag an error, or perhaps ask for confirmation of a file overwrite. If MC doesn't show that message to the user, and to do so it would need to understand that there is a message to display and an action to be taken, then the Handheld Sync may just hang, or stop when the message timed out, or never complete. I don't know how WMDM devices are supposed to respond to that situation, or if they have flexibility in how they respond, and I don't think I shall research that. JRiver added de-dup capability to Handheld synchronisation for some reason, after all. I trust they had good reason.

I've actually got a handheld definition right now which duplicates files every single time I run it.  I wonder if this behavior I'm seeing (file level duplicates) is related to this decision to do Handheld Sync de-duping.  In my opinion it would be smarter to have file level duplicate code that has the option to overwrite or ignore.

File level de-dup may make more sense, as long as it also took into account the destination directory. That is, it would be full path level de-dup. Otherwise files of the same name, but belonging to completely different albums, would be removed from the synchronisation. I think we could all find songs with the same file name in out collections, even though they belong to completely different Artists and Albums.

But file level de-dup would take away from the power of using tags, and possibly force people to rename files to force a track to synchronise.

Nothing like a bit of robust discussion in the morning hey Brian? Now I need some breakfast! 

[Gatherum]

Not everyone curates their music collection carefully, or their Playlists.

No, but MC is one of the few music jukebox programs catering hard to those who do. I, for one, wouldn't want to see an MC that follows the Microsoft philosophy of assuming that users are dumb and break nuts.

[blgentry]

Yes I thought that through as well, and you are correct. But if (proper) de-duping reduced the amount of files to be copied over, at the simple level that would reduce the time it takes to synchronise, which I for one would appreciate.

Unfortunately JRiver doesn't have any control over how a handheld device responds when something tries to copy the same file to it twice, into the same location. They have to support as many devices as they can, and can't know how each will respond.

Let's just pretend that we are copying to a file system for a moment to simplify the discussion.  If that assumption falls on it's face, that's fine, but I think we need to start there in order to work through the logic path you have started.

In the best case, which is when the handheld is treated by the PC as a hard drive, Windows will rename the second file by adding " (1)" to it. I saw that yesterday, but I was writing to a hard disk folder.

I'm pretty sure that the (1) behavior is done by MC.  I've seen it in Rename, Move, and Copy scenarios.  I've also recently seen it in Handheld Sync.  I do this on a Mac, and I'm pretty much dead certain that OS X does not silently rename files.  I'm willing to bet all the money in my wallet that OS X just lets files with the same name be overwritten.  As I've done with "cp" for years on every flavor of Unix.  (OS X is a flavor of Unix at it's core.)

File level de-dup may make more sense, as long as it also took into account the destination directory. That is, it would be full path level de-dup. Otherwise files of the same name, but belonging to completely different albums, would be removed from the synchronisation.

Yes.  All I'm saying is this very very simple case:

MC's Handheld Sync is told to write file X into a directory.  File X already exists.  MC should handle this.  It should either:

1.  Silently overwrite the file.
2.  Silently not overwrite the file.
3.  Have options that control 1 or 2 above, including a date stamp comparison.

It makes ZERO sense to produce X(1) .  When would that ever be useful in Handheld Sync?

My problem handheld sync which you referenced should be explained.  It is not driven by multiple playlists trying to write the same thing.  Rather, it occurs after file X has been written by Handheld Sync.  Then later, Handheld sync is run again, with the exact same setup and it tries to write X again, producing X(1).  This bug happens to only occur with files with accented characters in their names.

I can run sync, the recheck and immediately see that it wants to write these files again.  This is some odd bug that *might* be specific to this particular handheld (Ipod Nano 1G with rockbox).  I should probably do some testing of this same data set, pointed to a naked folder on my local disk and see what happens.

I'm not feeling super great right now, so please take any short sentences or conclusions here as the ramblings of someone who isn't happy or feeling well.

Take care.

Brian.

[RoderickGI]

Yep. When I said Windows I should have said WMDM (Windows Media Device Manager). But it probably is MC doing the renaming, as it certainly silently does that renaming.

I agree it doesn't make a lot of sense to rename a file with " (1)" during a Handheld sync. Except that in doing so, all files get copied to the handheld, and the Playlist created by the process works for all files. It is sort of a fail-safe mode of operation.

The "Recheck Sync" function seems to force copying of the files over to the Handheld device again, even if it is already there. It probably shouldn't do that, particularly as it copies the file over again and renames it with a " (1)", but I believe that is by design. In fact, with all my playing around, I now have up to eight copies of some files in my Handheld Device directory on my drive! Note that the Playlist on the device always refers to the last file copied across, so it would always work.

If you just run Sync, then files aren't copied over again of course, since MC already knows they are on the device. Glynor and I had a long discussion about sync lists and such a while back, and how they work isn't exactly obvious. The wiki says; "If you've set up any Playlists to sync, you can also force Media Center to re-build the Sync List by clicking on the Recheck Sync button in the header of the view." In practice that means that the files will be copied over again, even if they are already on the device. That is the way the Sync List works.

So, only use "Recheck Sync" if you want to force recopying of the files. For example, if you have deleted them from the device manually. MC doesn't check what is on the device, even a local hard drive folder, it just keeps track of what it believes is on the device. If nothing has been deleted from the device externally to MC, then just use the Sync function, not the "Recheck Sync" function.

If you are only running the Sync function and are getting second copies of the file with the " (1)" suffix for files with accented characters in their names, then that does sound like a bug. But it would be a strange one, as MC would first have to think the files was new, and hence needed to be added to the Sync List, and would then have to recognise it was the same as a file already synced to the device, and hence rename it. Worth testing in a clean manner.

I knew you weren't feeling well. I hope you get on the road to recovery soon.

[DJLegba]

We're getting off topic here. Time for a JR rep to chime in with "yes, it's a bug, we'll fix it in the next release".

[Gatherum]

We're getting off topic here. Time for a JR rep to chime in with "yes, it's a bug, we'll fix it in the next release".

This is what I've been waiting for.

[Gatherum]

Nothing? D:

[Gatherum]

Ping.