There are many things that could account for this, and you provide very little info about what you're actually doing. Start by picking a single reference bluray disc and analyzing/documenting exactly what you're doing.
First, as tij questions, are you bitstreaming the audio in JRiver? If you are, MC does not alter the audio from the file. Conversion to PCM does not make the sound better or worse; the PCM sound is identical to the sound decoded by your DENON, but converting to PCM opens the door to JRiver or the Denon modifying the sound differently.
How you're ripping in MakeMKV could cause it. You might be ripping a different audio track than your hardware player is using. If you are ripping multiple audio tracks in MakeMKV, JRiver might be selecting a different one from the file than the one the hardware player selects. You need to verify exactly what audio tracks you ripped into the MKV, which one MC is selecting, and which one the hardware player is selecting. For all we know, your hardware player is playing the lossless track on the bluray, but you ripped only a lossy track into the MKV.
You understood yourself you can have different Denon settings for different inputs. But Denon and Marantz units can also apply different settings based on audio format, regardless of input. If MC is decoding the DTS HD-MA lossless track to PCM, the Denon might do different things to the audio just because it is PCM and not DTS. You'll need to check what settings the Denon is using each different format is playing.
There are some advanced AVR settings, like Dynamic Range or Dialog Boost, that may only get applied to encoded formats like DTS/DD, and are not applied to PCM. If you're using these that could affect the sound, so if you want to match it you'll want to bitstream in JRiver.
Finally, some hardware players have options that modify the picture or sound to "enhance" it. Check your player; a difference in sound quality could be due to that.
The sound you get from a MKV rip played through JRiver will be identical to what you should get from a hardware player, but IF AND ONLY IF:
- You are playing the same audio track
- Your receiver is applying the same processing
- Your hardware player is NOT processing
- You are bitstreaming in JRiver, or disable JRiver options that modify the sound if converting to PCM.