Thanks, gentlemen.
I have tried every permutation of "specified output" and "specified output when necessary," mp3 and m4a both checked and unchecked (when using "specified output when necessary," as it's grayed out with "specified output"), and "output format" in DSP Studio both checked and unchecked.
The results are 100% consistent:
(1). I cannot play lossy file types at all unless I exclude them from DSP processing. More specifically, (a) when I have "specified output" checked or (b) when I have "specified output when necessary" checked with lossy file types not excluded, I cannot get them to play at all. I get five seconds of silence on the progress bar and then nothing.
(2). In contrast, when I have "specified output when necessary" checked with lossy file types excluded (unchecked), I can get them to play, but, obviously, not as DSD.
(3). Lossless does play as DSD when the renderer is bound to the DSP DLNA server, regardless of whether format is "specified output" or "specified output when necessary."
(4). Checking or unchecking "Output format" in DSP Studio doesn't alter any of the preceding results.
This all tells me that JRiver actually hasn't figured out how to upsample mp3 and m4a to DSD or there's a bug showing up in my particular usage.
The only devices I have on the network that can handle native DSD are Sonore Signature Rendu SE/Wyred 4 Sound 10th Anniversary DAC and Sonore UltraRendu/Topping D90 combinations. The results are consistent between both renderer/DAC combinations. The two Sonores use basically the same board, so perhaps there is something about those components that makes JRiver unable to feed upsampled lossy.
On the other hand, I have just re-verified that, with either renderer/DAC set, mp3 and m4a do play as DSD when upsampled to DSD64 in Audirvana and Roon. (They also upsample to DSD128 and DSD256 with Audirvana and Roon. I haven't yet tried DSD512, however, out of fear that I will cook my CPU. DSD256 is a pretty heavy processing load.)
I really would like to get JRiver DSD upsampling to work with my entire library, especially when you get up to DSD256, at which the Topping D90 really shines. If I can make that happen, I'd abandon both Audirvana and Roon and would convert my NUC to a dedicated Linux JRiver server box rather than run JRiver on my desktop Dell as i do now.