I am experiencing similar problems. I am running Windows 8 using my TV. I am able to playback various video files like mkv in J. river Media Center 18, but I am unable to playback vob files. I can view the first few seconds and the application crashes.
I've been trying to configure the settings like you advised, but I need more specific directions since I'm not able to make it work from what I've been doing. I find the DSP Studio and output format, and then what do I do in there??
Welcome. A few things to try:
1) Make sure you're using Red October Standard in Options > Video
2) Disable any bitstreaming in Options > Audio
3) Set DSP Studio > Output Format to a simple consistent configuration like 2 channels, 44,100 Hz. If that works, then explore more complex arrangements.
Let us know how you fare.