I have no issues even with my GT 960.
The tricks to get it work:
- Update to latest Nvidia drivers
- Don't touch Nvidia control panel about output colors or level modes. Leave them by default, and let the TV accept low/limited range (16-235). Only test those if you want to experiment or to solve issues.
- Update madvr and lav (optional).
- Leave HDR disabled in Windows 10.
- Check that lav video HW decoding is set to DXVA2 or D3D11, and most, if not all, codecs HW Accel enabled.
- In madvr, in the display HDR page set it to let madvr decide.
- (optional) I personally unchecked exclusive mode (it isn't needed and it adds some annoying display switch) and I reduced the queue sizes to mínimum in processing and to only 2 in "windowed mode", since my 960 is only 2Gb DDR5.
- (optional) I let JRiver change refresh rate to fit the content but I leave always 4K resolution. My 960 is still able to perform nice luma upscaling with lanczlos or jinc (If I remember well). You can upscale chroma with less demanding algorithms.
In my experience is working great for SDR and HDR content switching always to the correct signal/mode to the display. When I push stop while playing HDR it comes back to SDR.