Does the uDAC5 support native DSD playback and/or DSD playback via DoP?
If you can't get DSD playback working, you may have to check the DSD bitstream in DoP format checkbox in your second screenshot. Otherwise, it looks fine to me. Just make sure system volume and MC's volume are set to 100% when playing back DSD files.