This happens to me sometimes (with a fast video card, processor,etc) especially when I've recently imported any files. It also happens at times in standard view while I'm browsing the Video view scheme. At the same time there will be several instances of "FFV" shown in my windows tool tray. Is this the same for you?
I have a feeling in my case it is trying to build thumbnails or something, but sometimes it takes a very very long time to get back to normal speed. I wish there was a way to see what was going on in the background (i.e. if files are being imported, thumbnails built, tags updated, whatever). CNTRL-ALT-DEL shows that the "JRworker.exe" (or something close to that) process is running in the background, I wonder what that does? It gets frustrating when you don't know why everything is so slow.