There are a few things that could cause this.
I've gotten that message before and the following fixewd it. for me.
Start with Output Format in DSP Studio. Set the output encoding to None. Under Channels set it to 5.1 (or however many speakers you have). I know you're thinking, "What about 2 channel stereo tracks?". Media Center will only play them on the left and right speakers and silence in the others. Mixing to JRSS mixing (recommended). Check, "For stereo sources only, mix to 2.1"
In the sample rate box, I have outputs 88,200Hz and above set to 96,000. My decoder for some reason won't work at 88,200 and you can either choose 44,100 there or 96,000. Mine works that way.
If those settings don't work for you, this is the area to change things one at a time until it works, depending on your setup and your saying your system can handle it.