When I set the "Output Encoding" to DSD, my A/V receiver (connected via HDMI) shows "PCM 176,4 kHz" on its display. But there is only white noise coming from the speakers and you can hear the music in the background very quietly only.
This is a common mistake. This setting is NOT the one you use to stream DSD files to your AVR. It's used in unusual circumstances only; like with DACs that can not decode PCM and can only accept DSD. You want this setting set to "None".
So are there other setting in JRiver I have to check or is it just my A/V receiver which can not handle DSD?
If you want to send DSD files to your AVR as DSD to be decoded natively, you want MC to "bitstream" your DSD files. But, as Hendrick indicated, DSD over HDMI isn't a common thing. So it probably won't work.
The good news is, there are at least two ways you can listen to DSD files using MC and your AVR:
1. Disable output format as above. Then just play your DSD files. MC will convert them to PCM on the fly and send them to your AVR. DSD converted to PCM generally sounds very good.
2. Connect to an external DAC, or your AVR via USB. If your AVR can act as a DSD capable USB based DAC, then MC should be able to bitstream DSD to your AVR's DAC over USB. Setting bitstreaming to DSD (in Audio options) should make that happen.
I'd be inclined to just do #1 above and be done with it. It's pretty likely that your AVR's DAC converts to PCM internally anyway, so there probably aren't any real sonic benefits of sending pure DSD to your AVR. You'd have to research that to be sure; just throwing that out there.
Brian.