Anyways you should set Output Encoding back to None in this case. In MC's Options > Audio below the Audio Device section there's a Bitstreaming option, and it should be set to DSD. If it says None then that means DSD bitstreaming isn't enabled and there's your problem. Here's an image with the option highlighted in a red box (taken from Windows but the option is there regardless of OS).
Thanks for all that info! Yes, the DAC supports native DSD
Well, it specifically has to support DoP (DSD-over-PCM) so it works on macOS, besides native DSD which macOS doesn't actually support. It's a bit complicated compared to Windows (and Linux). Since the Output Encoding option for DoP seems to work you can just ignore that part.