Well, there's part of your problem, DoP is the only one supported on macOS.
Make sure DSP Studio > Output Format > Output Encoding is set to None. Also make sure both internal volume and system volume is set to 100% otherwise bitstreaming may fail. If the issue still happens after ensuring those things, you may need to contact the DAC manufacturer.