Everyone who has posted to this thread is trying to help you, out of nothing but altruism. We're not JRiver employees, and we're not responsible for your problem. You might want to keep that in mind.
Using multiple zones is often a solution for getting around finicky hardware issues. And video files often require different output settings than audio files do. Evidently you are not experienced with those sorts of issues. That's ok, but you'd be well advised to keep a more open mind. Getting computers to work with some formats can be tricky, and solving problems requires flexibility, since there's no "Make it work" button. I understand that you assert MC is to blame for this. But the reality is that MC doesn't decide what audio formats are supported.
Your video card (the HDMI source) handshakes with the Oppo (the HDMI sink) and they negotiate what audio formats are acceptable. Then, the video card tells MC what audio formats are allowed. Period. MC has no input in that process. You're getting the error message you got because MC tried to output in a particular format, and the video card rejected it. Considering your comment about the Denon receiver, I think that something that may be eluding you is the fact that every device in the HDMI chain must support a format or feature for it to function. You keep talking about the Oppo as if that's all that mattered, but that's only half the story. The other half is your video card and driver, which you've never mentioned.
The fact that the behavior persisted with another sink means that the issue could be with the video card or driver.
Since you don't understand why the product offers "source number of channels", I'll explain: It's to do one of two things: to allow you to intentionally reformat the audio stream (as in upmixing), or to work around the quirks of different output devices. For example, I have seen many output devices that do not support 3-channel audio. It is a perfectly valid format, officially recognized by HDMI, but some devices don't support it. Some devices don't support 5-channel audio (as in 5.0, not 5.1).
So a lot of people are able to leave it at "source number of channels". But it's very common to have a device that doesn't work properly with one of the unusual formats, so what all those people do is select 5.1 or 7.1 (no upmixing/downmixing) and then MC sends the extra channels as silence. That works for almost everyone. If I play a 5-channel DSD file on my system over HDMI, the video card rejects the format, and I get the same error message as you. I set the # of output channels to 5.1, and the 5-channel DSD file plays fine as 5 channel. So it works for me too.
But not you.
So here we are. I hope that explains it for you.
Regarding 64bits, the reason I brought that up is that HDMI doesn't support 64-bit audio. So I don't see how you're ever going to be able to output at that bit depth across HDMI. MC can process internally at 64bits, but you can't output at that depth, it has to be downconverted. Perhaps MC got the error before it got that far. If I were sitting at your computer, I would know the answer, but we have to depend on you to provide helpful information. If you look at the Bit Depth field for that file, I imagine it will say 1 for the file that gave that error.
Try the other choices in Audio device settings for bit depth. 24bit integer in a 32bit package is a likely suspect, but try them all on that file that threw the error message, and see if any of them work. It depends on what your video card driver wants. If none of them fix the problem, it is probably the channel count that the video card objects to.
What does "same or worse" mean regarding the Denon test? With Output Format=5.1 channels, all your stereo files still play as silence?
And as I said before, I suggest you try another player app that can use WASAPI, like foobar, to verify that the problem isn't in your video card driver software. See if you can make it play a 5-channel file without converting to 6 channels (5.1). If you can't then you know it's your video card that won't let "source number of channels" work.
Regarding Kal's comment, Kal is an experienced user who tests lots of different equipment, so he has encountered equipment that doesn't support various formats. That comment is exactly what I would expect someone like that to say, because you can never be sure what formats will work with a piece of equipment until you try it. Your implication is you think his comment means the "source number of channels" option in MC is broken. But it does exactly what it's supposed to do.
Another thing to look at: the Analyzer in DSP Studio. You say that if you select output channels =5.1, then your 2-channel files play only silence. Ok. So let's take a look at the Analyzer module of DSP Studio, while one of your 2-channel tracks is playing with output channels =5.1. Show us the whole DSP Studio window so that we can see what modules are checked. Looking at the analyzer module, it will show you traces, and numeric volume levels across the top. What do those show when you're playing your stereo track that you hear as silence?