Hi zeltak;
I did have problems trying to get 5.1 Surround Sound through the single SPDIF coax cable; that is, until I got my wonderful Logitech Z-680 5.1 speaker system.
The Logitech Z-680 speaker system has a control panel with Dolby Pro Logic II & dts Surround Sound hardware built-in. This is similar to your Yamaha.
My Logitech manual recommends for best sound with 6 speakers (Dolby 5.1) to use the single digital output cable form my sound card. That is the jack that is marked as Center channel. Make sure you run the Creative Surround Mixer & put a check in the Digital Output Only box; this will convert the Center channel output jack from an analog to a digital out jack with full 5.1 Surround sound. You will then use only the one (Center channel) jack output to your Yamaha.
When hooked up with an analog output (separate Front, Center, & Real output jacks) the Center channel is simply one of the analog outputs (Center only). The Creative Surround Mixer does not have a check in the Digital Output Only box; this will leave the Center channel output jack as a simple analog Center out. This is the default out-of-the-box setting unless you changed it to Digital Output Only.
I have my DirecTV satellite audio output (Dolby 5.1 Surround Sound) going through my Audigy sound card by way of the ATI A-I-W 9600 Pro video card with TV tuner.
I also have the Optical output from my DirecTV satellite receiver going directly to the Optical input of the Logitech Z-680 control panel.
I can switch back & forth between the direct Optical input coming from the DirecTV & the coax input coming from the Audigy; the sound is the same.
I always get the full 5.1 channels (left front, center, right front, left rear, right rear & woofer.)
When DirecTV is broadcasting a show in Dolby 5.1, I will get the full 5.1 Surround Sound effects.
One note about the digital coax output from the Audigy. If you use the single digital coax out (Center channel) the Creative Surround Mixer Test button will only test the Front Left & Front Right speakers. If you use the analog out (all three jacks) the Test button will then send the test signal to all 6 speakers. Not really sure why this is but suspect that the test signal is geared to analog & does not work fully on digital.
To make sure all is working with the digital setup go to the Creative folder, find & play one of the 5.1 ac3 test songs such as Samba Del Gringo.ac3. It will rock your system with real 5.1 Surround Sound. You will simple hate normal stereo from then on. Also note that MC will not play the ac3 music so you will have to use the Creative player for ac3 music. Hope JRiver comes up with a solution to this one of these days as ac3 blows everything else away.
Hope this all helps a little.