Madvr has a slight favor for Nvidia (Full screen window mode is only available on Nvidia, for instance)
As I understand it, this is because AMD don't support the features required for Windowed Overlay mode, for example. Similarly, older AMD cards have trouble with DXVA2 Copy-Back.
Madshi seems to strive for feature parity as much as possible when developing madVR.
Another option is to get a fast card that is not passive and replace the cooler for an aftermarket passive one. This voids warranty though but if you're absolutely set on getting everything out of Madvr and get a perfectly silent setup, this option might be worth investigating. I have no idea how big of a card you can get with a passive aftermarket cooler though, you'll have to do your own research. You also have to consider your case as a passive cooler in a small closed system might still overheat if there isn't sufficient fresh air flowing past it.
Honestly, I'm not a fan of completely passive systems. I love the idea of being able to build a PC that doesn't have a fan or a spinning disk inside it - but I've never seen a passive system that doesn't throttle its performance due to heat build-up. If you have a passive heatsink on the CPU, and a passive graphics card, then you need a fan inside the case moving air over them anyway.
The only real exception to this, is if you have a system that is only being used as an audio server, because that's only going to be using a small fraction of any modern CPU. (say 1%)
It's too bad that the desktop Haswell chips don't have the faster GPUs available on the mobile chips. Those
might have enough performance for Jinc 3 scaling.
The GPUs on the desktop chips definitely aren't fast enough to use that on everything though.
I can't say what the minimum GPU is for madVR to work at its best though.
I like the look of the new ASUS GTX 670 DirectCU Mini for an m-ITX build, but it's almost certainly not a quiet card. (and overkill for madVR - at least for now)