Something strange is going on, I am encountering a similar (but not identical) issue, with the same setup (I let JRiver toggle HDR, since I don't want any tone mapping when playing SDR content).
RTX 3090 -> Denon X2800H -> LG 65C2
On any driver higher than 546.65, Audio passthrough is messed when playing 2160p23 at 12 bits. By messed I mean no sound at all or a blip of sound every 10 seconds. The AVR display switches back and forth between Atmos and DD+. There's clearly something wrong with the signal.
Same issue with different players (MPC BE/HC, VLC, JRiver), different renderers (madvr, MPC Video Renderer, JRVR, whatever VLC uses).
One time I even rebooted Windows and everything nvidia related had disappeared (like, no NVCP in right click menu on the dektop, no system tray icon)
8 bits works, And, weirdly, 60 Hz / 12 bits with passthrough works too. I probably should try more combinations (HDR/SDR, 1080p, etc.) but since driver 546.65 work perfectly fine, first thing that comes in mind is a driver issue. And that sucks, because I use the very same PC to play games and can't afford to NOT update my drivers for latest games...