Topic: Cross Thread Hang on Windows (per crash info in event viewer) (Read 18824 times)

whoareyou · « **on:** November 21, 2018, 08:55:01 am »

I originally posted this in the media Network area, but getting no responses over there. Hopefully someone will take notice of this post.

Original post subject is ----> DLNA Playback Dynamic Zone Hanging JRiver
https://yabb.jriver.com/interact/index.php/topic,118238.0.html

Summary is that the MC client and server end up in deadlock (per event viewer in windows). I originally thought this was only occurred with the MC windows client, but it just happened with Gizmo.

I believe this issue can be reproduced by disconnecting the dynamic zone from the server while the client is still connected to the server. (In case it's important my dynamic zone is a OPPO 205). Sometimes it seems to occur when the server has been put into standby, and the client is still running.

It's intermittent issue that seems to trigger with a couple of different scenarios, but when it occurs the UI's are completely hung, and the application (client and server) are unresponsive. The common theme seems to be one of the components, server, client or zone, fails to reconnect and MC components are deadlocked.
MC components need to be stopped and restarted.

This happens with the latest build, and the last official released build. I also tried a fresh install on a completely different computer, and have experienced the same issues.

It seems to have started with the latest builds.

Thanks

In each case both client and server need to be completely stopped and restarted to resolve the deadlock issue.

Will someone at least acknowledge this post? I'd like to get it resolved.

JimH · « **Reply #1 on:** November 21, 2018, 09:32:25 am »

There is a DLNA change coming in the next build that may affect this.

whoareyou · « **Reply #2 on:** November 21, 2018, 10:12:12 am »

Quote from: JimH on November 21, 2018, 09:32:25 am

There is a DLNA change coming in the next build that may affect this.

Thanks. Is that build 60? I've tested on that build, and have same issue.

bob · « **Reply #3 on:** November 21, 2018, 03:46:25 pm »

What do you do to “disconnect the zone from the server” exactly?

4 jumping cats · « **Reply #4 on:** November 21, 2018, 03:49:10 pm »

I guess he/she unplugs the playback device.

whoareyou · « **Reply #5 on:** November 23, 2018, 04:05:12 pm »

It's an oppo-205 being used as a renderer. I simply turn it off between listening sessions.
The computer is also generally in standby.

It *might* be more reliable when the computer hasn't recently come out of standby, but not positive.

I still think that it's the client triggering all of this.
Also, if I first start playback from the server, and then use the client controller, I believe that keeps everything working correctly,
I'll keep trying to narrow it down, but it's intermittent, and I forget sometimes to pay attention after it hasn't happened for a little while.

I've also seen the oppo displayed on the clients after it is clearly been powered down. Clicking on that zone most certainly hangs things.

cncb · « **Reply #6 on:** November 24, 2018, 09:14:02 am »

I also keep getting (roughly every week) the application hanging while using a remote control app. MC becomes unresponsive and right-clicking the server tray icon does nothing. I have to kill both exe's in the Task Manager to recover. I don't see anything in the Event Viewer for MC, though.

bob · « **Reply #7 on:** November 26, 2018, 08:44:42 am »

If the rendering device is doing what it's supposed to do, powering it down (and not just pulling the plug) should cause it to send the BYE-BYE notifications that remove it from MC's device tree.

If the renderer is removed by pulling the plug or it doesn't respond because it's locked up or temporarily off the network, MC should time out on the command send when you try to play to it and it should be removed from the tree.

whoareyou · « **Reply #8 on:** November 27, 2018, 11:52:48 am »

Yeah, I'd expect a timeout at some point, but I never see it. Perhaps I don't wait long enough for the timeout? How long should the timeout take?

Either way, it's a very bad user experience when this occurs, and from my perspective there's something funky going on with the threading.

Luckily I can work around it, but I'm hoping your team can figure something out.

Also hoping I can figure out a consistent set of events where I end up with the cross thread deadlock. I'll post again if I learn anything new.
Thanks

bob · « **Reply #9 on:** November 29, 2018, 11:07:10 am »

Quote from: whoareyou on November 27, 2018, 11:52:48 am

Yeah, I'd expect a timeout at some point, but I never see it. Perhaps I don't wait long enough for the timeout? How long should the timeout take?

Either way, it's a very bad user experience when this occurs, and from my perspective there's something funky going on with the threading.

Luckily I can work around it, but I'm hoping your team can figure something out.

Also hoping I can figure out a consistent set of events where I end up with the cross thread deadlock. I'll post again if I learn anything new.
Thanks

No more than 30 seconds for the timeout.

If you can find a consistent way to reproduce it I'd really appreciate hearing about it!
Thanks!

whoareyou · « **Reply #10 on:** December 04, 2018, 04:05:22 pm »

Did some more organized testing....

On Server:
When playing directly from server to Oppo, everything worked as expected.
Zone always displays properly, depending if Oppo is turned on/off, and playback is as expected.
Turning oppo off and on, never caused the server to hang (without client in the mix).

On Client:
Zone is not propagated from server to client correctly unless the zone existed when the client connected to the server.
If the oppo is turned off while the client and server are both active:
Server immediately removes the Oppos as a dynamic zone
Client never removes the dynamic zone (this will eventually hang things)

If the oppo is turned ON while the client and server are both active:
Server immediately adds the oppo as a dynamic zone
Client never displays the zone

Restarting the client properly updates things and gets zones in sync with server. Seems like there's some communication not taking place between server and client.

Because I turn the Oppo off, and the server is suspended at times, I think the client's zones not being updated is causing the hang. I seem to never have hangs if I make sure to always start the client only after the oppo and server are both up and running.

Anything I can look at on my end such as network? Firewall? What's the protocol for server sending this info to the clients?

I think everything is ok (based on other forum posts), but never hurts to double check.

4 jumping cats · « **Reply #11 on:** December 04, 2018, 04:18:33 pm »

You could always force a dynamic zones refresh by clicking the button in the right click menu. (Although I know this doesn't obviate the fact that this should be happening automatically)

whoareyou · « **Reply #12 on:** December 04, 2018, 09:43:05 pm »

That's an interesting suggestion. I didn't realize that option is available.
Will be interesting to see if that "fixes" things, or I end up with the client and server both hanging, and requiring a restart.

whoareyou · « **Reply #13 on:** December 11, 2018, 02:46:05 pm »

Ok, I took a look at this and the option to refresh on the client is greyed out.

I still believe the overall issue with client and server hanging is that the client and server are not properly communicating with each other after a renderer has disconnected.
The JRiver server side recognizes the OPPO 205 being disconnected and reconnected. The updated status is not being reported to the client.

It seems to be either a network issue or a software issue (or both since the deadlock should timeout).

What can I check on my network that's related to how the server is communicating to the client?

bob · « **Reply #14 on:** December 13, 2018, 04:15:30 pm »

Quote from: whoareyou on December 11, 2018, 02:46:05 pm

Ok, I took a look at this and the option to refresh on the client is greyed out.

I still believe the overall issue with client and server hanging is that the client and server are not properly communicating with each other after a renderer has disconnected.
The JRiver server side recognizes the OPPO 205 being disconnected and reconnected. The updated status is not being reported to the client.

It seems to be either a network issue or a software issue (or both since the deadlock should timeout).

What can I check on my network that's related to how the server is communicating to the client?

Keep bumping this and I'll try to get to it when possible.
The connection to the library server is a static one time only thing so that it won't reflect changes to the renderer list on the server side.
There needs to be a way to handle this, for certain a timeout would be completely necessary.

whoareyou · « **Reply #15 on:** December 16, 2018, 08:19:47 am »

Thanks. I'll keep bumping.

whoareyou · « **Reply #16 on:** December 27, 2018, 03:10:48 pm »

Bumping as requested. Thanks!

RoderickGI · « **Reply #17 on:** December 27, 2018, 06:36:21 pm »

Quote from: whoareyou on November 23, 2018, 04:05:12 pm

The computer is also generally in standby.

Just to muddy the waters, the computer (I assume you mean the server above) going into Standby (Sleep) will cause what your seeing very easily. At least it used to, which is why my HTPC isn't allowed to sleep these days. Well, that and because I can't send Wake On WAN (Wake On LAN requests from the internet) through my modem/router from the internet. I would like to let it sleep again, if the issue below was fixed.

How to reproduce:
Set the MC Server to sleep after a short period. Let's say 30 seconds, just for testing.
Start MC on the Server.
Start a MC Client and connect to the MC Server.
Wait one minute for the MC Server to sleep completely.
Try to use the MC Client. It will lock up, requiring a kill and restart, which will then send wake commands to the MC Server, and the MC Client will connect.

I believe that happens because the MC Client isn't notified that the MC Server has gone to sleep, and can't see that it has. So it keeps trying to connect, and locks up. I used to have this happen all the time, as I had the MC Client up but not playing anything, and then would like to check something on the HTPC Server, not knowing if the Server was asleep or not. If it was, the Client locked up.

At one time this was going to be addressed so that the MC Client would recognise that the MC Server was not available, and send Wake On LAN commands to the server. I don't believe that has happened.

So, not quite the same issue, but the same in that the MC Client doesn't receive updates about the status of the MC Server. This may explain some of the random nature of what you observe.

whoareyou · « **Reply #18 on:** December 28, 2018, 03:10:41 pm »

Thanks. Yeah that could definitely be part of the issue. I'll have to test that scenario out and see if I can replicate it.

bob · « **Reply #19 on:** January 04, 2019, 08:54:59 am »

Quote from: RoderickGI on December 27, 2018, 06:36:21 pm

...
How to reproduce:
Set the MC Server to sleep after a short period. Let's say 30 seconds, just for testing.
Start MC on the Server.
Start a MC Client and connect to the MC Server.
Wait one minute for the MC Server to sleep completely.
Try to use the MC Client. It will lock up, requiring a kill and restart, which will then send wake commands to the MC Server, and the MC Client will connect.

I believe that happens because the MC Client isn't notified that the MC Server has gone to sleep, and can't see that it has. So it keeps trying to connect, and locks up. I used to have this happen all the time, as I had the MC Client up but not playing anything, and then would like to check something on the HTPC Server, not knowing if the Server was asleep or not. If it was, the Client locked up.

At one time this was going to be addressed so that the MC Client would recognise that the MC Server was not available, and send Wake On LAN commands to the server. I don't believe that has happened....

This shouldn't happen any more, the client does send WOL commands now when it errors out on a sleeping server.

whoareyou · « **Reply #20 on:** January 04, 2019, 12:57:53 pm »

Thanks for that info.
I'm probably not going to do any further testing until I see an update specifically for some of the issues I've experienced and discussed in this thread.

I'll bump the thread every once in a while.

RoderickGI · « **Reply #21 on:** January 04, 2019, 03:38:12 pm »

Quote from: bob on January 04, 2019, 08:54:59 am

This shouldn't happen any more, the client does send WOL commands now when it errors out on a sleeping server.

It seems I missed that change, or forgot about it. I see it was early in MC24 development. At least I think it is the below. I shall try it out.

24.0.5 (3/13/2018)
3. NEW: Playback from library server will issue a wake request before starting playback to wake a sleeping server.

bob · « **Reply #22 on:** January 04, 2019, 03:52:18 pm »

We did some more testing this afternoon.
The WOL works great.

However if you don't have your computer setup to do WOL or you are not on the LAN with the server you will get an apparent hang.

It's not really hung, it just has a long timeout on each main thread screen redraw because it's checking the connected state of the remote server.

We've played around with solutions like caching the last connected value for N times which makes the UI somewhat usable.

Pondering having the client reload the main library after some number of failures to get the connected state. The downside is that bringing up the server would no longer result in the client operating as if it was still connected. OTOH, it can't really do much in that super slow mo situation even if we did let it ride.

Thoughts?

RoderickGI · « **Reply #23 on:** January 04, 2019, 07:32:54 pm »

I did a couple of quick tests here.

On my LAN, with the server set up correctly for WOL, the current solution is almost perfect.

When the Server has gone to sleep and the Client is still open, the Server is woken silently and the Client does what it was asked to do, all be it with a slight delay. Around 15 seconds in my setup. There is no apparent loss of context at all. I had a look at the underlying local copy of the library, Delta, Server Delta, etc. and only a few files had recently been changed, with those associated with the task I was doing; playing a playlist.

The only improvement to that which might be of value is if a message popped up saying "Waking the Media Server from sleep", as some people would start clicking randomly if MC didn't respond almost immediately.

I think if the connection cannot be re-established because the Server doesn't wake after multiple attempts with the Client connected, then if trying to reload the Server Library would result in a better chance, try that. I'm not sure this would have any better chance of waking the Server though. I suspect that would have the same result. The next step would be to try restarting MC, which would again send WOL commands and load the Library defined by the Startup Options. I think MC will already switch to the local Main Library if it is restarted and fails to connect to the previous library, when that library is a Server library. If none of those steps gets a functional MC back, then something is seriously wrong. Either the Server is locked up, or there is a network problem, or similar. Not a MC issue.

I would be happy with the above steps, even if they took some time, as long as I was informed of what was happening. Perhaps a message sequence along the lines of;

"Waking the Media Server from sleep" with no Cancel button.
"Server didn't wake. Trying to reload the Server Library." with a Cancel button.
"Server didn't wake. Trying to restart Media Center using default library" with a Cancel button.
"Server didn't wake. Trying to restart Media Center using Main (local) library" with no Cancel button.

With such messages being displayed, users are much less likely to start clicking around the interface to try to get a response, or kill MC thinking it has locked up.

I am not interested in ever seeing the non-responsive, looks like it hanged, or only somewhat usable UI, ever. If the above sequence failed, I need to fix something and don't need MC to limp along with partial functionality.

As always, it is dangerous to ask me for my thoughts! But maybe the above feedback could improve the user experience. Thanks for asking though Bob.

whoareyou · « **Reply #24 on:** January 05, 2019, 04:31:28 pm »

I think this thread is going off course from my original issue with the cross thread hang.
Of course UI should never look unresponsive, and I'd welcome that being corrected.

There's still issue with the dynamic clients refreshing properly between client and server.
I don't want to lose track of the issue this thread was created to address - the cross thread hang.

Thanks

RoderickGI · « **Reply #25 on:** January 05, 2019, 05:20:30 pm »

Sorry. Our bad. Maybe some nice Administrator will split our comments off to another thread.

bob · « **Reply #26 on:** January 05, 2019, 06:31:27 pm »

The dynamic clients not refreshing is actually what I used to cause the condition for testing.

Any solution should encompass that issue as well and as RoderickGI proposed above with the "Server didn't wake. Trying to reload the Server Library." with a Cancel button." step would also address the dynamic clients issue.

whoareyou · « **Reply #27 on:** January 06, 2019, 08:12:13 am »

Thanks for that additional information. Quite honestly, I probably encountered the issue discussed in that last thread entry, but conflated it with other stuff going on between client / server/ OPPO.

I appreciate everyone's assistance with this. I think between everything discussed, the UI will end up with a nice improvement.

whoareyou · « **Reply #28 on:** January 21, 2019, 09:45:16 am »

Bumping as requested.

bob · « **Reply #29 on:** January 21, 2019, 02:02:43 pm »

It's in the queue, thanks for the reminder.

whoareyou · « **Reply #30 on:** January 22, 2019, 08:25:41 pm »

Thanks for the update. Any idea of how far down in the queue? FYI - It's not critical to me, but I am curious

INTERACT FORUM

Author Topic: Cross Thread Hang on Windows (per crash info in event viewer) (Read 18824 times)

4 jumping cats

4 jumping cats