I can't help you with your playback problems (it's probably DirectShow filter related though) but if you use AutoGK to convert the files from raw VOB's to XviD-compressed AVI's you'll have better luck playing them back (and much more room for more movies since it squishes them down signifigantly).
It's really easy to use. You basically just rip the DVDs onto your hard drive using DVD Decrypter in IFO Mode (or something that works similarly). Open AutoGK and open the main movie's IFO file as the
Input File (it will also work with VOBs, M2Vs, and bunches of other MPEG-based content but the IFO is the easiest to use and still get all the audio tracks and whatnot). Then you select what size you'd like the output to be (I find 1024MB gives pretty good quality for most ~2hr movies), select what audio tracks you want included (and Subtitles if you want them) and hit go.
You can
download the latest, completely free, version of AutoGK here. (The site seems to be down right this minute, but it was working yesterday and I'm sure it will be back soon.)
There's a nice guide for using it here.To make sure the playback of these (and many other) video files works well, I'd recommend you install the latest version of the
ffdshow DirectShow filter (get the Alpha 2004-10-12 version) and
AC3Filter.
(BTW, for anyone who wants them,
the "Daily" builds of ffdshow are available here. I haven't tested them thoroughly enough to recommend them, but they are built by the actual developer.)