Don't use DSP Studio > Output Format > Output Encoding to playback DSD, as it'll convert everything including PCM to DSD.
Instead you should enable DSD bitstreaming, and that should do the trick.
https://wiki.jriver.com/index.php/DSDThere is a few things you should know, one thing you should know is if your DAC supports native DSD playback or DSD-over-PCM (DoP) or if it supports both of them. I going assume it's going to be both here, but I don't think you mentioned which DAC you're using.
Anyways DoP is dependant on the max PCM sample rate of your DAC, which means with DoP and a max PCM sample rate of 768 kHz with your DAC you're going to be limited to DSD256 via DoP. To playback DSD512 you'd have to use native DSD playback (on Windows likely via an ASIO driver) and not DoP. Another thing is that if you're using macOS it has no support for native DSD playback via CoreAudio, so you're going to be limited to DoP and DSD256 there regardless what you do. On Windows the DAC manufacturer likely has an ASIO driver available, which should allow for native DSD playback at the full DSD512 after switching to the ASIO audio output for your DAC in MC's audio options. Linux might work too with native DSD, assuming the Linux kernel you're using has support for your DAC.
On Windows if you're using WASAPI for audio output, it's limited to DoP only. As mentioned before, native DSD on Windows requires an ASIO driver provided by your DAC's manufacturer and set as the audio output in MC's audio options.
Here's a quick cheat sheet for DSD via DoP and what it requires:
DSD64 via DoP requires a PCM sample rate of
176.4 kHz.
DSD128 via DoP requires a PCM sample rate of
352.8 kHz.
DSD256 via DoP requires a PCM sample rate of
705.6 kHz.
With most DACs out there, this would be the limit of DoP right now. For
DSD512 via DoP the PCM sample rate would have to be doubled again,
1411.2 kHz.