If I'm not mistaken, indeed the current max limit for Mac is 4X DSD (aka DSD256) via DoP.
But then again, 8X DSD (aka DSD512) on MC for Windows is only available for native DSD output when using output format, and not DoP (which is what the MC for Mac needs and uses for DSD). If it's not available for DoP, then 4X DSD is likely the limit there. My guess is the PCM sample rates in current DACs don't go up high enough for 8X DSD via DoP support?