Any idea what this could be? Is it a hardware limitation? The machine is pretty low end but Kodi can handle 1080p on a 1080p TV/projector:
64GB Crucial RealSSD
AMD A6 5400K Black Edition CPU
Asus F2A85-M LE, AMD A85X
4GB Corsair DDR3 XMS3
Zotax GT 730 2GB GPU
400w PSU
Think this is a hardware issue and would the GTX960 save the day? Would it also be worth upgrading the CPU to an A10 5800K?
I can't give you a definitive answer on that, because so many settings affect video performance, and I'm not quite the expert in that area. If you don't get tearing with Kodi, perhaps you should try settings similar to Kodi and see if the problem persists. You should look at Video Clock, Hardware Acceleration, De-interlacing, and Options/Video/Display Settings in particular. As you are using RO STD, right click on the screen while a video that exhibits the problem is playing, select DirectShow Filters, then Enhanced Video Renderer, as see if the video is dropping any frames, that the frame rate matches the source file, jitter, and any other issues.
On the hardware, what I can say is that it wouldn't be a good idea to install a 4GB graphics card into a Pc that only has 4GB of memory. Windows is going to reserve a big chunk of the PC's memory for hardware, so you will probably see problems. In fact, you should have a look at the Windows Resource Monitor Memory page, and see how much of your 4GB of memory is already "Hardware Reserved". You may be surprised.
So if you are going to install a GTX 960, you need to upgrade your memory as well. Maybe you could get away with 8GB, but if the pricing isn't too different, I would go to 16GB. In fact, I have 16GB in my HTPC just so I don't hit problems when I install a discreet video card into it.
Of course, if you are using a 32bit version of Windows, you aren't even getting the full 4GB of memory. You will need a 64bit version of Windows.
Then you need to start thinking about how much power you have. Upgrading to 16GB uses just 4 Watts more power. Adding a GTX 960 adds another 72 Watts. Upgrading to the A10 5800K processor adds another 32 Watts, for a total requirement of 277 Watts, and a recommended power supply of 327 Watts. You are still okay with your 400 Watts PSU then, but it had better be a good quality PSU, or you may start to see troubles.
Those power estimates are based on just the components you listed above plus mouse and keyboard, and the "eXtreme Power Supply Calculator" at
Outer Vision. You will see that the calculator recommends a 430W power supply with 16GB+GTX 960+A10 5800K. I haven't included any fans, USB devices, water cooling, Wireless, TV Tuners, or anything else.
Note: I have the registered version of the "eXtreme Power Supply Calculator", but I have no affiliation with the supplier. I have found it useful for years though.