Your chosen settings are probably too high. If as you say they are interlaced videos, they are going to be 60 fps after deinterlacing, which means every video frame has 16.6ms to render - the stats say yours take 20-24ms.
Switching from FSRCNNX 16 to 8 might already do it, i reckon, with a minimal impact on quality only.