I don't have that motherboard, so I don't know what incompatibilities/limitations the audio section might have. You could have a driver issue. However, there are two versions of the audio chipset on that motherboard; one supports DTS and the other does not.
I don't know what you mean by "Tests in compatible settings via Win10 confirm DTS output and reception from B&K". You might need to explain that more precisely, if none of the following work...
In the Windows control panel sound section, check the properties for your SPDIF out, and ensure that DTS Audio and Dolby Digital are both available and checked, and that you have the correct sample rates underneath checked.
Have you been able to get any non-JRiver software to send DTS through you S/PDIF port? Try VLC. (JRiver has a score of options that can make or break this; VLC as two.)
VLC has an option (in the AUDIO settings) for HDMP/SPDIF audio passthrough. Set that to enabled, and set your device appropriately in the setting right below it. If you do that, and VLC refuses to send DTS through your optical out, you either have a driver issue you might have the non-DTS chipset, and you better check your motherboard documentation to make sure. If you have the non-DTS motherboard, then your only option would be to use the HDMI out (with JRiver doing the decoding), which means you're not going to be using that B&K... The only other workaround I can think of is to set the JRiver output format to Dolby Digital; then the DTS will get converted to DD, which already works for you. But then you'll need to setup a separate video zone in JRiver so that your regular 2-channel audio doesn't also get converted and compressed.
I will say that B&K is too old if you're going to be serious about video. It doesn't have any multichannel inputs other than the TOSLINK, and it cannot decode any of the newer audio formats such as are to be found on blu-rays. Maybe it's time to move on...
If VLC is able to send DTS, then JRiver will be able to as well...
First, check and report the exact audio format of the DTS files you're having trouble with; check the "Compression" and "Channels" fields. If it's DTS and the channels=3, 4, or 5 then your audio driver might be barfing on uncommon DTS formats, and there probably isn't a fix for that other than the one I already gave you. Some audio drivers think multi-channel always means 2-channel or 5.1 (6-channel) and they don't work properly with other uncommon channel counts.
To best work with what you have in passing DTS, I suggest adjusting the following in JRiver:
-Make sure your "Bitstreaming" option is set to S/PDIF
-Under Audio Device (in JRiver settings) try the ASIO or WASAPI options you might have there.
-There's a setting "Play silence at startup for hardware synchronization"; set it to 1/2 second.
-In DSP Studio, make sure "Output Format" is turned off (unchecked)
-In Options->File Types, check what you have for "DTS Audio (dts)"; you might need to try different playback methods under that option. You might also have to try variations for the Matroska Video option in the same section.
If VLC works but none of this gets it done, you might have an issue with the JRiver audio/video filters, and one of the people here more expert than me (like Hendrik) might have to step in to save you.