For audio device try to pick your device with (WASAPI)
Next Select the Output format box so we can get all 6 channels, it may be processing the audio as Stereo, but you should be able to tell if this is happening or not.
So if you are not bitstreaming, then JRiver is doing all the processing with the signal before it goes to the receiver.
Your other application may be doing bitstreaming and your receiver is doing its DSP on the stream.
For a quick check you could try bitstreaming (HDMI), if it sounds better, turn bitstreaming back to none and go into the DSP settings again and select the room correction and set up your speakers there.
Jrivers DSP engine is usually much better than any receiver. If you have a direct though mode on your reciever I would use that and set everything up in JRiver.
Let me know how it goes. I'm no expert but will to get ya past the basics of the program the best I know.
Here is a good read.
http://wiki.jriver.com/index.php/Audio_SetupKen