Actually, you can do better than that, if you're willing to try something clever.
You got the error because you tried to play a DTS or AC3 source, with bitstreaming off. S/PDIF can't handle unencoded multichannel audio. Bitstreaming passes encoded audio (like DTS) through MC unaltered. That is why volume control does not work with it: MC is not allowed to touch a signal it is bitstreaming. But we can get around that; there are actually a couple of methods.
First: a new feature was added a while back that allows MC to manipulate a bitstream, but ONLY for volume...
See the little speaker icon to the left of MC's volume control? Right click it, and you will see an option for "enable volume when bitstreaming". Try that, with bitstreaming ON in audio options, and see if it gives you the results you want. It will let you adjust the volume of the bitstream, and nothing else.
Make sense?
The Second thing might also interest you: You can go even further though, and be able to apply equalizer settings, effects, and everything else, even for your surround sound.
What you need is multiple zones, and Zoneswitch.
So ignore different physical devices for the moment; I'm just talking about outputting to your Yamaha.
Here's what you do:
You need two zones: call them Yamaha Audio, and Yamaha Surround.
Yamaha Audio, you setup exactly as I already described, WASAPI and no bitstreaming. DSP Output format should be as you had it in your screenshot: empty.
Now Yamaha Surround, you setup differently. Still WASAPI, still internal volume, still no bitstreaming. But you activate the Output Format module, and you change your "Output Encoding" to Dolby Digital.
Here's what happens then with the Yamaha Surround zone: when you play a surround source, like your DVDs, MC will actually decode all the audio to PCM internally, in 64bit so the quality is perfect. It will then process the audio however you say, such as with volume leveling, or an equalizer, or with your manual volume adjustments. Then, it will re-encode the audio as AC3 Dolby Digital and shoot it out the S/PDIF to your Yamaha. You will get surround sound with volume control.
The catch is, you either have to remember to switch zones manually when you play content, so that you play audio content to the audio zone, and DVDs to the surround zone... OR, you setup ZoneSwitch, with a simple ruleset to do exactly that thing, and MC will play the right content to the right zone automatically.
Read here for more:
https://yabb.jriver.com/interact/index.php?topic=76605.0Why not just the surround zone and omit the audio zone? That would work, but EVERYTHING would be encoded to AC3, including your regular 2-channel CD quality audio, and that would degrade the quality of your CD music. Having a separate audio zone prevents that from happening.
Make sense?
So if you're game to embark on the brave new world of a more advanced configuration, you can have everything you want.