Thanks for the suggestions.
I've been at it over the weekend, I checked the steps from the jpg, I found that on avs forums too with a lot of discussions about configs and suggestions. I tried most suggestions and config examples there too. The only thing I hadn't done is the nvidia output in 8-bit. To confuse things a little, that was with my laptop with nvidia card, I sold the laptop over the weekend and I am back to the latest generation i5 NUC. It has the same issue so it must be my TV somehow.
Its definitely not the movie thats intentionally being too dark. Its immediately visible, from the intro paramount mountain top or that gold theater intro. Some have that bad robot intro, its really obvious and just waaay too dark with absolutely no details whatsoever in the dark parts, its completely crushed. Also, those movies would look washed out, they look dull and dimmed and low on contrast in bright colorful scenes. Its really not a matter of the scenes being intentionally dark.
Whatever I try to get proper HDR ouput, the screen remains too dark for almost all HDR material. Its all the movies that are too dark, they are all 4k/UHD with HDR, but some/most maybe all HD/SDR series come out too dark as well (havent tested everything). On the other hand, one series would play fine which was also 4k HDR. Just one, so I wonder if it has anything to do with how they are encoded that my tv doesn't handle? Maybe in the hdr metadata? I have no idea really.
Anyway, I did find a working solution and that is to disable HDR completely, in Windows settings, and configure Madvr to do tone mapping to 100 lut and output in SDR. The hdr logo no longer pops up obviously, I increase gamma on the tv to 2.4, set black level high and play with brightness to get black to be black (which is around 54). This gives me the best detail in dark scenes but in the test screens (blinking black and white bars) the last 2 or 3 white bars are clipping. I'm not sure if I can get that right somehow, but black is more important to me than 2 shades of white clipping.