I compared a little more between how JRVR does HDRtoSDR vs HDRtoHDR TMing.
I see with HDRtoSDR even on scenes with low peaks it consistently shows spline TMing 203->203, and then as the peak increases above 203 but below specified display peak (say, 400, in this example) it will show something like 300->300, for example, and then if peak increases beyond display peak it will then show something like 500->400, for example.
But with HDRtoHDR it shows no TMing until suddenly when peak increases beyond display peak it will then show something like 500->400, for example. And because of this, we get this issue with a suddenly ugly jump in the image.
HDRtoHDR should behave exactly the same as HDRtoSDR other than any color space differences and obviously final output transfer function. But from a TMing perspective, it should be the same, and I believe that's the fix for this issue.
Though I agree there is likely a bug with HDR>HDR TM in JRVR as there is no issue with HDR>SDR TM, "203>203" and "300>300" with SDR content means the same as "nothing" in HDR. It simply means no tonemapping (brightness-wise). JRVR correctly doesn't touch the picture (SDR or HDR) from a TM point of view if peak in content < display peak. So I don't think this means anything.
There is no issue with HDR to HDR if the shot starts at a level that needs TM from the start, or doesn't need TM during the whole shot. For example, to keep your 400nits MDL example, if a shot ends with no TM (under 400nits), and the next shot start at 600nits, you willl suddenly see 600>400 when before that there was nothing, but JRVR will handle it propely because it's scene detection will tell it right away that tonemapping is needed from the start, and there won't be any sudden jump in brightness, even with HDR>HDR.
Again, this issue only happens when a shot/scene (as identified by JRVR) starts with no tonemapping (peak in content below display peak) and calls for significant tonemapping (brightness-wise) during the shot. Not sure why it's handled differently in HDR>SDR and HDR>HDR, but I agree it shouldn't be (from a brightness TM point of view).
Looking at the difference in handling should allow to resolve this issue, but what the OSD says in SDR is purely cosmetic and doesn't help much.
Hendrik, please could you confirm that you are looking into this? It isn't a case of handling a specific case that would cause issues elsewhere, otherwise we'd have the same issue with HDR>SDR when using the same target.