my 2 cents for what they are worth as I'm more of a musician that a techie ...
As Brian and Jim said the gain has to be identical ==> google an ABX freeware, and make sure the output is the same
As CSimon said no DSPs should be in the comparison (if possible) Now I'm saying if possible be cause I just went to their website
Software upsampling / downsampling with selectable algorithm up to 32-bit 1.536 MHz or down to 16-bit 32 kHz PCM (14 options, most "apodizing")
Delta-Sigma modulators for upsampling PCM/DSD content up to 49.152 MHz 1-bit SDM (11 oversampling options)
Selectable dithering / noise shaping algorithm (7 options)
Order of the settings from left to right; resampling/oversampling algorithm, dither/noise-shaping algorithm, output sampling rate and output sample type (PCM/SDM).
this is a lot of DSP choice off the bat ... some of these are effecting the signal before its getting to your DAC -- can't compare apples and oranges. If you are upsampling or oversampling using this HQ program you must also do so through JRiver .. essentially this is no longer bit-perfect -- whether or not it sounds better to you is something else. Same goes with room correction, convolution filters, parametric eq.
CPUprocessing, especially on a bit-perfect stream to a DAC, run in JRiver in exclusive mode won't have any effect (positive or negative) -- its coming out bit perfect right?. Saying that though, it has been often cited that tweaks to the Windows registers (like Audio Optimizer) aren't going to improve sound, and combined with JRiver it could hurt (not to mention these server tweaks running on core mode or whatever its called I've been reading about). So compare JRiver without that -- its designed to avoid what you are worried about.
Leaving any biases aside, if there is a significant difference in what you are hearing, IMHO, it will be coming from either a higher gain (volume) or from something different in the DSP chain (or both). Can't be sure, but looking at the website, I bet its in the DSP (what they are calling "selectable algorithms"). If you like their DSPs, I pretty sure you can get the same with JRiver other than the dithering choices. But I leave that up to the experts to choose for me.