JRVR does not use ICC profiles at all yet (its going to be added as one of the next features). If you do big corrections that would result in such differences, then that might be it. Although I don't think madVR supports ICC profiles either (unless thats a new addition in the beta versions), but I guess you made a 3DLUT of them to feed to madVR?
I'm not seeing any fundamental color differences here no matter what target I tonemap to, without involving an ICC.
Sorry I'm late back, but I finally have some extra observations. I'll number the paragraphs for easy reference.
1. My monitor is hardware calibrated, the ICC profiles it creates appear to be equal to generic standards (effectively passthrough to the monitor internal LUT), except for Native mode, so I can just script madVR to output generic built-in standards for BT.2020, BT.2100, BT.709 according to content and flick the monitor mode to match. I had to create a LUT for madVR to display correctly when the monitor is in sRGB, as there is no built-in option for that.
2. Now that JRiver has an ICC option, it appears that the current ICC profile is indeed being used. No matter which monitor mode I choose the output looks about the same within expected differences in brightness and gamut, provided I restart JRiver after switching modes. This is ideal behaviour, an advantage over madVR.
3. However, it still looks oddly over saturated compared to madVR, regardless of gamut clipping options. So, a glowing red light in JRiver is a much deeper red than in madVR. Which looks attractive until you realise Tom Hanks sitting in a lawn chair at night in Apollo 13 has a red shirt that is not dull deep red like a shirt in the dark would be as portrayed in madVR but almost glowing deep red brighter than other objects in the scene almost like a dull light source. I am not seeing any anomalies with general photo editor usage using the same ICC profiles, so I don't think they are broken.
4. A potential problem is that JRiver engine Ctrl-J always reports that tone mapped HDR is being output as BT.709 regardless of content, output and gamut mapping selections. So, even when the monitor is BT.2020, as prescribed by the ICC profile in use, JRiver reports it is outputting tone mapped SDR as BT.709. I am not sure if this is a conversion error (wide-gamut SDR is a desirable option that at least preserves one of the major benefits of HDR encoding) or a logging error (eg: variable in logging not being updated and containing the wrong value). It might imply that an unnecessary conversion from BT.2020 to BT.709 is in the pipeline even though the selected output is BT.2020 which may well clip and oversaturate some colours.
5. In all cases, madVR tone mapping HDR to SDR looks closer in appearance to direct HDR passthrough in PQ mode, provided you disable the brightness tweak (by placing a named empty file in the madVR) to aim for pure BT.2390, especially if you disable dynamic peak measurement. So, if HDR tone mapping in JRiver is more saturated than HDR passthrough to a PQ monitor, something feels a bit off. JRiver BT.2390 is also higher contrast and more "pop" than madVR, where BT.2466 is slightly lower shadow contrast than madVR (I can't find any information on which standard is considered better or for what context, so this setting is hard to choose objectively, but I can see the third option is popular with gamers though).
6. There appears to be another tone-mapping problem with HDR in that the JRiver engine peak nits setting is not behaving as expected. If someone is tone mapping to view in the dark at lower brightness (say, because their IPS monitor has poor black levels and they want to sacrifice some "pop" for deeper blacks), they may want to set peak nits to match their SDR monitor running at 100-200 nits backlight. madVR allows you to do this and the HDR highlights roll off smoothly, even with no options enabled to recover explosions and highlights. With JRiver engine, any setting below 203nits clips highlights badly. So in Star Wars III, the opening battle scene pans down from the scrolling text looking directly into the sun: madVR reproduces the sun as a bright natural featureless glow at all peak nit settings, but JRiver depicts the sun as a jiggling cluster of pure white overlapping rectangles at peak nit settings below 203 nits.
7. Part of the problem faced is when two products potentially use potentially different methods to achieve the same goal with limited information available, it is hard to know which one is more accurate. Perhaps one clips gamut and the other does a perceptual shift, but how to tell? Even so, as mentioned before, madVR tone mapping seems perceptually closer to PQ HDR passthrough and JRiver is not just higher contrast and more saturated, but there are sometimes also scenes where it feels like the colours don't quite match with each other into a coherent whole (usually darker scenes). Yet, the extra apparent contrast is quite pleasing and sometimes madVR looks a bit weak in contrast for darker scenes.
8. On the list of suggestions, it would be cool if one day JRiver engine could have an option to set the monitor to match the frame rate of the source. It seems to currently convert everything to the current desktop refresh rate. That option seems to improve clarity in madVR (ability to see very fine detail like pores in skin, individual hairs, etc). Also it is really handy that madVR can pass HDR directly through to the monitor without the Windows HDR desktop being turned on (HDR Desktop causes unacceptable problems, such as SDR content not being displayed correctly with proper colour management, more stress on monitor back light for extra brightness never used for the bulk of normal desktop tasks, less deep blacks). Early days yet, more will come I'm sure.