Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[marko]

There's something not quite right with library syncing from clients at the moment.

It seems that if I play and rate a ton of tracks on the server while the client is sleeping, when the client awakes, instead of getting the new rating, playcount and last played data from the server, it pushes its out of date data to the server instead, thus wiping out all that work on the server.

That's twice now. It's not good.

-marko

[Hendrik]

Are you still using the main version on the client, or did you update both now?

[Arindelle]

There's something not quite right with library syncing from clients at the moment.

It seems that if I play and rate a ton of tracks on the server while the client is sleeping, when the client awakes, instead of getting the new rating, playcount and last played data from the server, it pushes its out of date data to the server instead, thus wiping out all that work on the server.

That's twice now. It's not good.

-marko

I noticed that too .. luckily for me most of my "critical tagging is client to server. On this version what triggers the sync "event"? Most of the time it seems that it is on track change ... but on client playback this is not always the case (last song on album I've noticed, short songs). Any changes in the syncing when no playback is going on while your tagging ... is syncing time based btw?

On client => I didn't notice this last version, but the "new" tag window takes time to open. 15-20 seconds! This is painful. Old tag window <1 second, basically immediate. Opening new tag on server though is more or less immediate. Running Win10 on all machines now.

AND here is a positive observation   ... installation of WDM driver is very fast now which is good (or is it now not updating the driver unless the driver was modified which would be good too?). Still have to manually choose the driver on one machine, but not another ... but that is not new of course.

[Hendrik]

I actually found (and fixed) a rather bizarre problem in our handling of the Library/Merge and Library/Get MCWS command, but it would've affected the main build as well. I'll continue to investigate.

One thought, in the process of testing these new builds, maybe you restarted your server more often than usual?
If you restart your server while a client is "asleep", it may throw off the sync logic, since the "delta" calculation is then reset. I'm not sure this should really result in data loss, as the delta calculation from clients seems pretty solid, but its at least a direction into which we can look.

In general, clients never push their entire database to the server, they keep a delta of things that changed and only push this. Individual fields for individual files, no bulk transactions really.

Also, can you arbitrarily reproduce such data loss? If so, a detailed explanation of the steps would be appreciated.
I tried to, given the rough explanation you have given, but nothing happens.

And just to be sure, can you confirm with 100% certainty that this doesn't happen with the main build? I honestly still find this hard to believe. Behavior changes are rather unlikely. But I'm keeping an open mind.

[marko]

I put MMXV on the HTPC too.

I'll set about this over the next day or so, and post any findings in a new thread. Here's a quick overview of how I spotted this...

I'm playing and rating tracks in the car, using JRemote on iphone. (Playcounts only update if you allow the track to end normally, skipping to the next track also skips the playcount/last played update on the current track)

The playlist in JRemote is smartlist generated based around a search on "r=[]" (The number of tracks returned goes down as I work my way through them)
This morning, I parked the car on track 30 of 1122, so 1092 remaining.
When I got inside, server MC confirmed this to be so.
I ran the update to build 26 on the server.
I woke the HTPC to run the update there (via Remote Desktop)
About three days of tagging was then wiped off the server library. (I restored from this morning's 3am backup which reset the above smartlist to 1122 tracks)
I grumbled a bit. Posted here. Then went to bed

And just to be sure, can you confirm with 100% certainty that this doesn't happen with the main build?

No, but I'll check it out. Will take a day or so, will start a new thread if I find anything useful. I can say with 100% certainty that I've never bumped into this before (apart from the well documented playlist issues) and after many years, my workflows are quite deeply ingrained, nothing has changed in that regard recently. Give me a day or so with it while I try to work out a way to reproduce that doen't involve you taking a 50 mile drive with JRemote

-marko

[glynor]

I actually found (and fixed) a rather bizarre problem in our handling of the Library/Merge and Library/Get MCWS command, but it would've affected the main build as well. I'll continue to investigate.

I've been sporadically seeing the results of this issue, as described by Marko above, before I ever installed the MMXV builds (or they existed). I never had firm proof of any specific issue, though, which is why I didn't report it.

But a number of times over, say the past 30 days (give or take), I've had this happen:
* I know I watched that episode of that show. We watched it last night.
* Watched status of the episode shows up empty. [Number Plays] is at zero, and it still shows in my "unwatched current shows" Views.

Now that others are seeing the same, I think I've been hitting this for a little while. It does NOT happen all the time, or reliably, though. It just seemed to happen sometimes...

[marko]

I followed the exact same steps moving server and htpc up to build 27, and yup, what else did anyone expect? No problems at all. The server machine even took a reboot before the htpc got woken up. All was good.

Something was up last week though, so for now, we'll all just keep watching til we spot the pattern. The client / server things are tough though. Sometimes, the htpc can sleep for several days. Would that make a difference?

[Hendrik]

Its a mystery. I'll have to re-visit this part of the code in more detail to figure out if there could be any conditions that would trigger old data being send over.
I could totally understand it "missing" an update, so if you watch something on the HTPC, it may fail to sync its changes back to the Server, like one of glynor's examples, but not overwriting tag changes you actually made on the server.

Investigation is afoot!

[Hendrik]

The next build will include a change that will force any clients to re-sync their copy of the database if the server was restarted since the last sync. Note that both client and server need to be updated for this logic to take effect.
Previously a restart of the server could result in the client thinking it was up to date (even if it was not), and then producing a much larger "delta" than it actually had.

At least thats the theory.

[glynor]

Nice. That seems to match what Marko was describing (and perhaps applied to my more recent issues too).

While you're "in there" any love for our old friend?

[Hendrik]

While you're "in there" any love for our old friend?

But only because its almost Christmas.
Changed: Playlist deletions get properly synced from and to Library Server clients.

Again a fix both server and clients need to be updated for to take advantage of.
Testing (with Backup) appreciated once its out. Just to make sure it doesn't delete too much. =)

What happened today when you renamed a Smartlist? Did it clone itself due to that?

[glynor]

But only because its almost Christmas.
Changed: Playlist deletions get properly synced from and to Library Server clients.

 

Yes. I believe that's right. There were a bunch of impacts:

* All [Playlists] Is Any type references in Smartlists and Views got broken.
* Deleting a playlist caused a new "empty" playlist to be created
* Renaming a playlist caused them to clone.

I'll surely test to see after I make a good backup.

[Hendrik]

* All [Playlists] Is Any type references in Smartlists and Views got broken.

This might not have changed. (quite likely not)

The new change will just sync over the removal of the playlist. If other things need updating at the same time, we'll need to special handle that somehow.
Playlist Renames are tricky, since the library sees them as a add/remove pair and syncs them as such.

Do the views etc actually fix themself on the server (ie. ignoring any potential sync issues)? Or is that a more fundamental problem?

Regardless, still an improvement to allow deletions without leaving behind the garbage, and renames not causing dupes, even if other rename problems are potentially still there.

[glynor]

Do the views etc actually fix themself on the server (ie. ignoring any potential sync issues)? Or is that a more fundamental problem?

No. Views get broken too.

The Playlist ID gets cleared, effectively breaking that rule. I assume this is because the IDs get renumbered somehow during the sync, and it breaks references (or something like that). But, what is obnoxious is that it happens even when you aren't touching the Client-side machines at all.

If you edit any smartlist or View that contains a Playlist Is Any rule on the server while a Client is connected (even if idle sitting out there somewhere) the next time the automatic sync happens, it obliterates that Smartlist or View.  If you don't touch them (other than to just use them) while a client is connected, though, they are fine.

To make any edits to these kinds of Views or Smartlists, you have to ensure ALL clients elsewhere are CLOSED.

[Hendrik]

I split this out of the MMXV thread since it applies more generically.

Did anyone get a chance to test the new improvements yet? I didn't notice any breakage - did anyone notice improvement?

[marko]

So far, I have not seen any old data coming in from the client overwriting new data on the server, and I tried as many different ways as I could think of.

I found a lot of smartlists were getting borked with the [playlist id] thing, but if I switched to the shortcut "p=playlist" then a custom entry is used in the wizard, and that never got broken. I'll put a couple back to full playlist id method and keep an eye on them....

[Hendrik]

I don't think any of those playlist id things are going to magically be fixed, the playlist ids are not synchronized between clients - just the playlists themself are.

[glynor]

I found a lot of smartlists were getting borked with the [playlist id] thing, but if I switched to the shortcut "p=playlist" then a custom entry is used in the wizard, and that never got broken. I'll put a couple back to full playlist id method and keep an eye on them....

Interesting.

Could you explain a bit more thoroughly?

[marko]

Morning glynor,

It was one of the first shortcuts I learned back in the heady MC v9.0 days.

I have a couple of playlists at "Playlists\Viewscheme Groups\Duplicates\" called "Hidden" and "Recycle". These are a crucial part of my whole library, as dicussed somewhere else many moons ago. I use them for duplicate management in a view scheme hinged around a pane that displays the playlist group. By selecting files, I can then use pane tagging to add or remove tracks from these play lists.

The idea here is that during random playback, I don't want duplicates playing. So, all the dupes (kept for whole album integrity) except the best or chosen one, go in the "Hidden" playlist. Files marked for deletion go in the recycle playlist. I don't want those in the random playback lists either. As these two are so crucial to how my library works, having the server blow up the playlist exclusions ever so regularly is not so good. These are used everywhere!

If, instead of using the wizard to glean the playlist ID's, I simply enter -p=hidden,recycle it enters those as a "Custom" rule in the wizard, and so far, these have never gotten borked.

There is a caveat though...

Imagine I had a playlist on the playlist root called "Hidden Gems" that had stuff I always wanted to hear...
the -p=hidden,recycle rule would unfortunately exclude that playlist too as it contains 'Hidden'.

The shortcut does appear to match anywhere in the playlist path though, so if I used -p=duplicates\hidden then the "Hidden Gems" list would not be excluded.

-marko

[glynor]

Brilliant and crucial info.

I do much the same as you with the Hidden files property, but I use special "magic Keywords" to manage it (in part due to this longstanding issue, though not entirely).

But I did not know at all that there was a way to do a contains style search against the Playlist.Name properties of the collection of Playlists. That's amazing and super-useful (and it makes complete sense that those will be unaffected by the sync issue). I'll convert my lists-referencing-lists over to this style tonight.