There really is no huge difference between using a custom LAV and the built-in LAV, assuming you use the same version in both. MC uses 99% default settings for LAV Video.
Disabled Hardware decoding obviously is the "None" hardware acceleration in LAV, so it should be pretty much identical in performance.
Enabled Hardware decoding in MC depends on your GPU, and RO Standard or ROHQ, since the different modes don't work equally well on all systems.
- In RO Standard, "DXVA2 Native" is used on all GPUs as the preferred mode
- In RO HQ, "DXVA2 Copy-Back" is preferred for NVIDIA and Intel GPUs, while "DXVA2 Native" is used for AMD (because Copy-Back is often quite slow on AMD)
So basically, I don't see how adding a custom LAV would resolve anything, unless the version you have installed in your system is actually a different one, and/or hardware acceleration settings are different.
That leaves madVR, which is actually quite more likely. We don't really support people manually upgrading madVR - its certainly possible to do, but we can't really offer support for that case.
I would recommend several steps to try to figure out what is actually at fault:
1) Change the advanced mode and only specify madVR as video renderer, not any splitter or decoder (it'll automatically use the built-in versions then), and test that config
2) Use plain RO HQ, delete the madvr plugins folder and let MC re-install the version its currently shipping with, and see if that resolves it
3) If even 2 above didn't help, then additionally go into the MC madvr plugins folder and execute the "restore default settings.bat" file. Note however that this will remove any custom madVR settings you may have performed.
If either one of those works, I would bet on a conflict in the madVR settings somewhere. Perhaps the version within MC you manually upgraded still uses the madVR settings file from the far older version we included with it - and as experience has shown, sometimes madVR doesn't really like running on old config files, especially when big changes to the options structure happened, like it did in 0.91
Upgrading madVR can be a bit finnicky at times, especially if you happen to be running two different versions, since madVR stores its setting in the registry so those two versions can collide and possibly make a mess of your settings.