To use DoP you need to set bitstreaming to DSD and use either WASAPI or ASIO. Either should bypass the Windows audio engine.
JRiver calls it "bitstreaming" but it is not bitstreaming in a way that bypasses the system volume control.
Windows doesn't know that anything is being "bitstreamed" it just thinks that there is 176.4 kHz PCM being played.
It sounds like there may be special handling for formats like native DSD or DTS-HD but that does not apply to DoP.
Many DACs link the ASIO volume level to the system volume level, so although it bypasses Windows, it's still affected by the system volume level.
Like Windows, the driver doesn't know that DoP is being played, it just thinks that 176.4 kHz PCM is being played. Only the DAC is identifying the DSD bitstream contained inside the PCM stream.
I don't think that WASAPI Exclusive disables the system volume control either. It bypasses things like the resampler but not volume control.
EDIT: It bypasses the mixer, but not the device volume. Assume that anywhere I said "system volume" I'm talking about the device volume that can be set in the system tray or the sound control panel.
At least with my system, using WASAPI with bitstreaming set to DSD works no matter what the volume setting or the WASAPI setting are.
Native DSD or DoP?
The actual system volume control, or JRiver's volume control?
JRiver's "system" volume control is not the actual system volume control.
JRiver's volume controls are disabled when it is "bitstreaming" but the system control is still in effect.
Since JRiver does not set the volume level to 100% when starting ASIO playback - the option only exists for WASAPI Exclusive - it means that DoP bitstreaming will not work if the system volume was set below 100%.
With WASAPI, enabling the maximize device volume option fixes bitstreaming. If that option is not enabled and the system volume is below 100%, bitstreaming won't work.
JRiver should be setting the device volume to 100% any time DoP or bitstreaming is enabled, whether the output is ASIO or WASAPI, and whether that option is enabled or not.
It doesn't matter for PCM playback, and it sounds like it wouldn't matter for native DSD playback, but it breaks DoP playback.
It would be nice to nail down exactly what settings fail for you. What volume are you using (Internal, System, Application) when it fails? Does using Disable Volume fix your problem? Does it matter if you check the WASAPI Maximize Playback Volume option? What DAC are you using?
It doesn't matter what JRiver volume is set to, it's the system volume level that breaks things, because JRiver is not setting it to 100%.