Running your display at 30hz refresh rate while the movies are encoded at 23.976 fps will cause this kind of stutter.
I see that 23/24hz aren't available in your list, which should be fixable:
1) Install latest AMD graphics drivers. See if 23/24hz are available now.
2) If not, open the AMD Control Center
3) Go to My Digital Flat Panels -> Properties
4) Uncheck "Enable GPU Scaling". Hit Apply. I found this setting interfered with being able to select certain refresh rates. See if 23/24hz are available now in MC18.
5) If not, go to My Digital Flat Panels -> HDTV Support
6) Check the box for adding 1080p24 support. Hit Apply.
7) To confirm what refresh rates are available, go to Desktop Management -> Desktop Properties. I have the following refresh rates available: 23, 24, 29, 30, 50, 59, 60. Changing the refresh rate here is the easiest way to determine what refresh rates your display will support (hopefully all of them!). Of course you'll want to retain 60hz as your default refresh rate and let MC18 handle changing the refresh rate at the start of video playback.
After all of these, you should have, at a minium, support for 24hz refresh rate. This should be fine when using the VideoClock option to scale native blu-ray 23.976 frame rate to 24 fps (this is what a hardware blu-ray player would do). If you have 23hz refresh rate support (which is really 23.976hz refresh rate), and it's supported by your display, that's even better because you don't need to use VideoClock. The refresh rate matches as closely to the frame rate (or multiple of the frame rate) as possible, the ideal scenario.
Here's a cheat sheet on the best refresh rates (from best to worst) to use for a video's frame rate:
23.976 fps (most films): 23,24,60
24 fps (some films): 24,60
25 fps (PAL broadcast): 50,60
29.97 fps (NTSC broadcast): 59,60
30 fps (random): 60