Okay, now we're getting somewhere. First I have to ask, why in DSP Studio > Output Format > Sample rate do you have it set to resample everything below 44.1 kHz and 176.4 kHz and above to 96 kHz? The DAC you have can output up to 768 kHz so that's why I'm asking, it seems like a bit of a waste to be resampling here, not to mention resampling is a lossy conversion. As a test, you might set all of these to None (and uncheck Output Format in the main DSD Studio list on the left too, just in case).
Second, you'll need to enable exclusive mode under Device settings by checking the
Open device with exclusive access option, as DSD bitstreaming otherwise won't work.
Third you can set Bitstreaming from None to Custom and in the Bitstreaming Formats window check the DSD box and set the drop down menu to
Up to DSD256 (4x).
Audio Path here is interesting though. Strangely (and this is the confusing part for me) is the
Downsample 8X for DSD to PCM conversion message. What's unclear to me is what does means, e.g. if it means you're converting DSD512 (aka 8x DSD) to PCM even though the input is a DSD64 track. Usually in that case it'd would make me think Output Encoding is set to DSDx8 but in the DSP Studio screenshot the option is set to None, so that's clearly not the case. Hopefully one of the devs can explain the
Downsample 8X for DSD to PCM conversion message here, as I've never seen that one. Anyways beyond that part, in addition it's resampling from 358.8 kHz to 96 kHz, because like first mentioned DSP Studio > Output Format > Sample rate is set to convert to 358.8 kHz to 96 kHz.
Here's a list of recommendations in simple terms to try to get DSD bitstreaming without any conversions working:
1) Make sure both macOS' system volume and MC's internal volume is set to 100%. Looks like MC's internal volume is already set to 100%, can't tell system volume in the screenshots.
2) Open up the Audio MIDI Setup app in macOS, and set it to use the highest sample rate your DAC supports, which would be 768 kHz.
3) Enable exclusive mode in the Device settings by checking the
Open device with exclusive access option, this is required for DSD bitstreaming.
4) Either uncheck Output Format in the DSP Studio list to disable it or go into DSP Studio > Output Format and set all Sample rates that are set to 96 kHz to None.
5) Enable DSD bitstreaming, by changing the Bitstreaming option to Custom then in the pop up window check DSD and set it to
Up to DSD256 (4x).
6)
This step is very important to pay attention to: Try playing back a
DSD64 track. You won't be able to playback DSD512 without conversion, but it can and should be able to bitstream and playback DSD64 tracks without issue.
That should hopefully do it. During playback another screenshot of the Audio Path would be very helpful here to verify playback. Or if any errors occur, screenshots of those may help too.
Good luck!