What DAC are you using?
For DoP to work with DSD256 on macOS (which is the only way to get DSD to work on macOS) your DAC has to support the PCM sample rate that is equivalent, which off the top of my head for DSD256 would be 705.6 Hz (unless I'm mistaken). For DSD512 it'd be double that, which I'm not aware of any DACs that support sample rates that high and as far as I know JRiver Media Center only supports up to DSD256 in macOS because of it.
Like I said before, native DSD playback isn't possible in macOS via CoreAudio, only DSD via DoP. Windows and likely Linux too (via ALSA if the Linux kernel supports DSD with your DAC) will support both native DSD if there's an ASIO driver available and DoP via WASAPI.
If your DAC doesn't support DSD256 via DoP in macOS, you'll have to either a) find DSD128 and lower versions of your DSD audio tracks or b) convert your DSD256 and DSD512 content to DSD64 or DSD128, which is a lossy conversion (and has to go through a PCM step in the middle of the conversion, as mentioned above). Personally I'd go with option a) and (hopefully) enjoy the music.
P.S. It worked if you converted (like in the screenshot above of Output Encoding) to 2xDSD aka DSD128 which is likely the DSD limit of your DAC. You won't be able to play back those DSD256 or DSD512 files in macOS without converting them or replacing them from the source(s) with a DSD64 or DSD128 version. I would
not recommend going back to using Output Encoding to convert everything to DSD128, as everything including PCM files will be converted too, and it's always a lossy conversion whereas the easier solution is likely getting DSD64 and DSD128 content if available to replace the DSD256 and DSD512 content.