I don't believe that SPDIF has enough bandwidth for 5.1. Do you have HDMI or optical connections available?
This is not correct. Optical
is SPDIF. There are two "types" of SPDIF connectors and cables, optical (TOSLINK) and coax, but they are effectively identical in function, and both can handle AC3 and DTS compressed 5.1 audio (Dolby Digital class, not Dolby TrueHD or DTS-MA class). The only functional differences between using optical cables and coax for SPDIF connections is that optical cables are fragile (stepping on one can ruin it) and length limited, but are immune to RF interference and ground loop problems like coax (and any electrical link). In practice, if you use a high-quality cable, coax is more versatile and convenient.
What do you have Tools -> Options -> Video -> Audio for Video -> Playback Device set to?
I have a problem with my setup where if I set this option to "Same as Audio" and the audio playback engine is set to WASAPI, then occasionally video playback will fail and I have to close and re-open MC to get it "fixed". This happens most often when I Skip Forward/Back through a video file a lot. Playing the audio for video back via DirectShow works flawlessly.
Otherwise, I'd try updating your audio device drivers. That sounds suspiciously like a driver issue locking up the SPDIF interface.