Ok did some experimenting today with my system, heres what I found when testing with a DTS-MA file on my system.
i7-4820k 3.7G
R9-390 Catalyst 16.6.1
Win10 Full updates all drivers current as of 6/8/16
32G Ram @ 2400mhz
Onkyo NR737(all firmware up to date) via High Speed HDMI w/Ethernet @ 5.1.2 capable(for testing, I Just ran 5.1)
Asus/LG/LG Eyefinity 3x 2560x1080(only the Asus via HDMI was used during testing)
JRiver is 21.0.83
WASAPI
Directsound seems to produce the exact same results.
Red Oct + No Bitstreaming - video/sound work, audio via PCM
Red Oct HD + No Bitstreaming - error
Red Oct + Bitstreaming - crashs JRiver/plays video, no sound.
Red Oct HD + bitstreaming - player plays but with 'waiting' msg, error prompt comes up, then promptly crashes should you click 'continue'.
Ideally, of course, running Red Oct HD + bitstreaming is the goal, so what am I doing incorrectly?