Encoding to DSD or DoP converts all audio being played to a DSD/DoP format for your DAC.
Some people claim it sounds better (it shouldn't) but it may be useful if you have a DSD-only DAC.
This is separate from DSD bitstreaming. Without DSD bitstreaming enabled, JRiver always converts DSD to PCM.
With bitstreaming disabled and encoding enabled, you have DSD > PCM > DSD/DoP.
The only reason you would want to do that is if your DAC only accepts a DSD/DoP signal, and you want to apply DSP to DSD files; e.g. volume leveling, room correction, eq etc.
Enabling DSD bitstreaming will always override encoding, and plays back DSD files natively without applying any processing at all - it bypasses everything.