M4A can mean ALAC files, or MP4 AAC files for example, so it might help the developers to know which one you're having problem with precisely.
Also, I just tested the converter, and it works to convert both ALAC and MP4 AAC file formats to FLAC. No idea why yours isn't. Maybe you need to download iTunes or something that installs the M4A encoder compatibility on your device.
As for DSD512, there seems to be an issue with the converter for ages now. There is no actual way of taking anything besides DSD64 and turning it into a PCM format of any kind (including FLAC). I've tested this with DSD64, 128, and 256 - sorry I don't have any DSD512 content and certainly no DSD1024 content, so it's almost certain these formats cant be directly converted to FLAC.
What is possible though, is the converter seems capable of converting DSD256 for example to DSD64, and THEN you can convert that to FLAC. Otherwise, whatever converter JRiver is using, seems not capable of doing the direct conversion. I know your next question would be wondering if doing multiple conversions would cause degradation to the file, to that, I can only say the following. If the converter is competent, then there should be nothing of the sort. Whether you can take DSD to FLAC without any theoretical loss in content is something WAY beyond my technical understanding to say definitively (but since they're both digital formats in reality, there shouldn't be an issue if the decoders aren't botched in practice).
In reality though, it truly doesn't matter because
- You have the original file, and the reason you're converting is due to size or compatibility with other devices, so it's not like you're losing your original file if you notice some insane loss in sound quality.
- The conversions I just tested where I did multiple conversions to the same file, and I couldn't notice an appreciable listening difference with casual listening.