I am not an upsampling believer, but I have played native DSD256 files in JR both as native DSD256 and as PCM176 via on the fly conversion, both without any issues at all, both in 5.1 by the way. I have not tried DSD512, since my DAC does not support it and there are few such native files available.
I use an older I7 machine with 16GB RAM and USB to my Exasound DAC via its custom ASIO driver. I don't think I ever needed to fiddle with buffering, etc. It works great for me.
So, playing the files is probably not the issue. Possibly, CPU resources for upsampling via JR are a problem, so you might first have to create the upsampled files in certain 3rd party tools designed for the purpose. Check CPU, etc. resources in Windows task manager.