So, in messing with the settings, you changed the Bitdepth from Automatic to "24bit integer". Your hardware doesn't support the 24bit integer format. You'd wonder why, because you looked it up and found your hardware & driver support 24bit audio. Probably what it requires is "24 bit integer (in a 32 bit package)". This is simply because your audio driver software only accepts DWORD (32-bit) data, and the hardware can only play 24-bit data.
Change the setting back to Automatic.
When you play 16-bit files, any unneeded bits will be zeros, so your audio data is unchanged.
Likewise, when you play 24-bit files, the extra 8 bits will be zeros, so your audio data is unchanged.
If there actually is an audible difference between 16-bit playback and 16-bit zero-padded to 24bit on your system, then either your driver software is processing the data differently (it shouldn't; take that up with Nvidia; perhaps an experienced Nvidia user can comment if this is a known issue, but I have never heard of it) or your Onkyo is processing it differently. Some receivers/processors (Marantz, for one) can apply different processing (room correction or DSP) settings based not on the physical input used, but on the detected data. So you might check all your Onkyo settings LIVE while 16 bit output is playing, vs when padded output is playing.