Welcome to the forum.
I have tried this before for ambio 4ch ("envelophonics"). It is not possible in JRiver, since JRiver only lets you use a 2 ch. VST dsp to process the front L/R channels. You can't use two instances of 2ch VST dsp to process both the L/R and SL/SR. I tried everything but JRiver, as far as I know, does not have this capability. What I ended up doing is using an external DSP (minidsp) to process the SL/SR channels.
More broadly, I eventually gave up this setup though. While ambiophonics is awesome for music, I found it very unsatisfying for movies. Two reasons (in addition to the setup difficultly mentioned above) for dispensing with ambio for home theater:
1. Despite ambiophonics' claims, dialogue from a split spatial source (even at 10* separation) is simply not as crisp as a single center channel.
2. Ambiophonics is designed for a single listener, max 2 listeners. Once you get out of the sweet spot, the advantages of ambiophonics is gone, in my experience. So, even on my small couch in my theater, the people to left and right of the sweet spot heard a very small sound-stage because the cancellation wasn't being applied to their ears.