You shouldn't use DSP Studio > Output Encoding > Output Format for DSD, it'll always do a conversion with a PCM step in the middle. Instead, you should really enable and use DSD bitstreaming when trying to playback DSD tracks and keep PCM tracks as-is because when converting to DSD there's always going to be a lossy PCM conversion step in the middle. Even with DSD to DSD conversions (e.g. DSD256 to DSD512) there's always a PCM conversion in the middle as well, so it's basically DSD256 > PCM > DSD512 in that case, and that's unavoidable.
https://wiki.jriver.com/index.php/DSDKeep in mind the DAC has to support DSD512 via native DSD in order to play it back. As far as I know, there currently isn't any DACs that support DSD512 over DoP because they don't support PCM sample rates high enough to support it. Usually with DoP DSD256 is the current limit, and for DSD512 playback you have to use native DSD. Now if your DAC doesn't support DSD512 at all (via native DSD), you will have to convert it in order to play it back, and that's also unavoidable. I'm going to assume since you were doing DSD512 (8xDSD) upscaling, that your DAC supports DSD512.
Now, since you're using an ASIO driver (as I don't have time to look up the spec sheets for your combo system to see what it supports) make sure ASIO is the selected audio output used in MC and that
Bitstream DSD in DoP is unchecked in MC's Options > Audio > Device Settings. This ensures native DSD is being used with ASIO.
Next go into DSD Studio > Output Format > Output Encoding and set it to
None, this is very important to avoid any and all conversions. With DSD bitstreaming it's sent off to the DAC as-is and it does the decoding and whatnot. Finally enable DSD bitstream in MC's Options > Audio > Settings > Bitstream option set from None to DSD. That should hopefully do it.
Also, when I set encoding to 4xDSD the sound is normal but when I set the encoding to 8xDSD the sound stutters and jerks, etc. Is 8x beyond the capability of my PC to process (I am using about 30% of the processor when encoding at 4x or 8x).
DSD conversions, especially the higher the DSD rate is (and if it's multichannel or not) are really the most resource intensive tasks Media Center can do, so this is exactly what it means. If you experience issues with playback like stuttering and jerky playback, this indeed means PC being used to convert whatever to DSD512 (8xDSD) isn't powerful enough to handle the on-the-fly conversion.