The issue is, macOS only supports playing back DSD over DoP so any DAC you have would need a PCM sample rate large enough to support DoP at whatever the level of DSD you're trying to use. If your DAC only have a PCM sample rate large enough to only support DSD256 (4xDSD) over DoP, then there's nothing that can be done about that. It's not a JRiver Media Center issue or something that can be 'fixed' in software, it's a limitation of your DAC and macOS. If you absolutely need DSD512 (8xDSD) then your only option is to use another operating system like Windows which can support native DSD playback via the ASIO driver you can usually find from your DAC's manufacturer's website.
For reference, DSD256 (4xDSD) requires a PCM sample rate of 705.6 kHz in order to use DoP. I've never seen any DACs that support PCM sample rates beyond that. DSD512 (8xDSD) over DoP would require the DAC you're using to support a PCM sample rate of 1411.2 kHz, which again I've never even seen one that does and not sure if they even exist.
