Sorry for the confusion. My original suggestion for setting Output Encoding was as a test, not for general playback. Output Encoding converts everything to DSD, including PCM. It also changes any DSD to the DSD setting in Output encoding. That is, if Output Encoding is set to 2x DSD then a 1x DSD is converted to PCM and back to 2x DSD. It should only be used if you want to play everything as one particular sample rate of DSD, which is pretty unusual. You stuttering is probable just the system converting everything to DSD.
If with output encoding off, dff file are not playing as DSD, then there is a different problem. You should look at Output Formating and make sure the higher end settings are set as No Change. If the greater than setting is set to anything else, then DSD will be converted to PCM to play. You should also set Bitstreaming DSD to on.
Try this. Set Output Encoding to None. Set all the sample rates in Output Format to No Change. In fact, uncheck all the options in DSP Studio. Turn Bitstreaming DSD ON. Play a DSD. Look at Audio Path (like box with 3 vertical bars on the top right). It should say something like DoP 5.6 MHz playing from dff or something like that and your DSD light should come on.