ok was still getting the mouse teeth. After much research I found that I need adaptive deinterlacing. For whatever reason I get adaptive deinterlacing and no teeth was from enabling 'accelerate decode as much as possible'. I tried the lav filter and the deinterlacing there but still got teeth, but not with ROHQ.