Ok, tried again today with my latest sources, and I still can't reproduce the problem. For me things work just fine, when starting playback either in windowed or fullscreen mode, either with FSE mode enabled or disabled, on all NVidia, AMD and Intel. Must be some driver issues you guys are having.
There are three NVidia specific 3D problems I noticed, though:
1) The monitor must be primary, otherwise 3D doesn't work at all. No such problem with AMD and Intel.
2) In windowed mode the present queue must be 4 frames or smaller, otherwise presentation times go through the roof. No such problem with AMD and Intel (or NVidia in FSE mode).
3) With 3D OS support enabled, 2D playback in D3D11 presentation mode crashes in some NVidia DLL. No such problem with AMD and Intel.
4) As soon as enabling 3D OS support, the GPU switches to frame packed output, meaning the display activates shutter mode. AMD doesn't do that, which I find better.
Generally my impression is that AMD is a more reliable solution for 3D playback at this point in time.