I did have full screen exclusive checked. When I unchecked it, then played a video, then hit the back button I got an entirely black screen. I was able to get to task manager, and saw that MC was "not responding". I killed it, and media server, then tried again, same thing.
Then I checked Tools > Options > Theater View > Behavior > Pause Video Playback when entering Theater View, which was also checked, so I unchecked it, tried again, same thing, black screen. (note - I was not seeing a black screen before, only the navigation page of Theater view, but unresponsive)
I restarted MC and MC Server, tried again; black screen.
I then changed RO HQ to RO Standard, and tried again; that worked. I went back to navigation screen and the video was playing in the window on the right, like I think it's supposed to. I clicked on watch, and the video went back to full screen, like I expected.
So, in my case, I had to make all 3 changes to 'fix' this issue.
I'd like to be able to use RO HQ; but it's not terribly important to me right now. Hopefully this at least lets JRiver reproduce the issue, and can fix it at some point in the future.
For those of us having the issue, at least it seems there is a solution; although not ideal.
Thanks mojave for providing the key!! This has been an annoyance for me for years, so I'm very happy to at least know how to fix it now.