I thought the AV receiver would output better sound using the SPDIF I always have movies encoded in DTS and the receiver always lit up so I knew what it was playing.
All things being equal, the sound from a DTS track playing via an 5.1 analog source should be identical to a bitstreamed DTS track. The only difference is where you choose to do the digital to analog conversion. Using S/PDIF, your AV does the decoding, and when using analog JRiver will do the decoding. Might there be some sound sound degradation along the analog pathway? Possibly, but I would argue that it's largely imperceptible. And what you lose in fidelity (if any) you gain in being able to use MC's DSP studio, videoclock, volume normalization, etc. The only reason I would go with S/PDIF over analog now-a-days is if you didn't have enough RCA cables or wanted to streamline the connection for ease of use.
Would the movies sound better played through 5.1 using the RCA cables?
If you are playing DTS and DD only, then no. However, if you are planning to play anything higher fidelity (DTS-MA, Dolby TrueHD, etc), you won't be able to play them over S/PDIF. You will have to let MC decode and send 5.1 analog.
The reason we dont play the music through zone 1 is hubby hates the presets in the AV receiver and liked to play with the equalizer in JRiver.
In truth we have never had really good music from this system.
We were thinking of getting a new AV
Using 5.1 analog will allow you to completely control your music via MC. In essence, under these conditions, MC is the true source of your audio. When you bitstream, MC is just a fileserver and your AV receiver does the heavy lifting. I trust MC much more than any stand-alone hardware.
As an aside concerning your AV, make sure that it is wired properly. Sometimes AV receivers (especially older ones) have strange wiring schemes (parallel connections) in order to get the proper speaker resistance. Some have speakers "A" and speakers "B" where one or the other is the high-powered connection and will not operate at full-capacity if the other is engaged. In this scenario, some speakers will be much quieter than they should be. Also check the ohms on your speakers (if they are aftermarket) and make sure they are wired correctly for the lowest resistance your AV supports.
Edit: I just want to stress again that S/PDIF had a time and place in home video, primarily in the 90s and 2000s to simplify connecting DVD players (and CD players in the audio world) to AV equipment without requiring a slew of wires. Those days are largely behind us. In lieu of HDMI, I highly recommend that you use the 5.1 analog connection, which is essentially future-proof for your current AV setup.
