This certainly sounds like a frame rate mismatch. Because you said that you used to get smooth movement before setting MC to change it, I assumed there was something wrong with the MC setup or the wrong rate was set in the tags for the movie.
I have ReClock installed and this is very useful for seeing what is going on. While the movie is playing, it can show the correct data for the movie and also the rate selected for the graphics card. I suggest you install this to get a bit more information.
Go to the SlySoft forum and look for the ReClock forum. The latest version of ReClock is available there. Install it and leave all the settings at default. In MC, choose Tools>Options>Video and under General video settings select Video mode: advanced custom. Under Custom Setting, at the top drop down, select Red October standard. Click the Add button then near the bottom, click the Type drop down and select Audio Renderer. On the Filter drop down, select ReClock. Now MC will use ReClock. Play one of your movies with MC windowed and look in your system tray for a circular clock face symbol. It may be red, yellow or flashing red and green. Click on this and a window will pop up. At top left, Video Stream, it will show the size, type and frame rate of the video. Across to the right and down a bit, Video hardware, Refresh rate, it will show the frame rate of the graphics card. If it is a 23.976 video, you want the graphics card to be set to 24. If it isn't, you will get jumpinesss. If the GPU is not 24, close MC and use NVIDIA to set the rate to 24. Now play the movie again and open the ReClock window and check again. If the rate is now correct, leave the movie running for a while and watch the System clock correction figure. This will gradually reduce. Ideally, this number should be at 0.17 or less to get perfectly smooth movement but it takes about 2 hours of constant play to get there. Even if you don't use ReClock, it may help find out what is going on.
Nick.