Good question. I am thinking about finally getting a GTX 960 very soon, probably an MSI GeForce GTX 960 Gaming 4GB. It would be disappointing if it couldn't play x265 well.
Which graphics card, Brand and Model, are you using?
As the problem is intermittent, check any files that are jittery and see if you can identify a pattern to those files. I don't have any 4K video at the moment, so can't compare, but look for attributes like frame rate, format, audio, etc. to see if any one type of file gives you trouble.
Also, the latest LAV filters and madVR are currently being updated to support 3D playback, so if you have been testing with the nightly builds you may see some anomalies. Using the latest public builds is best.
I doubt it has anything to do with make... They are all reference design and
I use drivers straight from nVidia.
As far as the video files, the intermittent issue is with the SAME files. I have a handful of 4k demos and sometime they play awesome, but other times at what seems like 12fps. And in fact, I haven't had any success at all recently with playback.
It looks like JRiver or Windows or the drivers are bypassing the hardware decoding altogether and just trying to do it in software. With that said, nVidia drivers are nuts! Since before Win10 was released. They only seem to care about gaming performance.
LAV filters were only recently updated. Prior, I installed then separately from what jriver installs/uses and had some success that way.
If there was a driver that worked, I dunno which one it was.... And it had many OTHER issues beside playback.
Very frustrating.
If AMD releases a video card with hardware decoding for X265 I'm buying it!