I would think you need a more than 10 channel output format (which results in a 7.1 mix target iirc) and I would stick to the usual 7.1 channel mappings but use the last 2 channels as your subwoofer output channels. I say more than 10 because you might find you need to have a few channels spare to use as temporary channels while you mix things around.
You would then have to do all the mixing using PEQ. You would use room correction, if you want, to set to delays amongst the main channels only (you could also do this in PEQ if you want).
The basic process is going to be something like
- attentuate the input main channels by the required amount (to be worked out later but remember to include the LFE adjustment)
sub summing for L
- copy the FL into tmp channel 1, apply HPF
- copy the SL into tmp channel 2, apply HPF
- copy the RL into tmp channel 3, apply HPF
- sum tmp channels 1,2,3 into Lsub channel
- apply delay to alignment Lsub output
- wipe tmp channels 1,2,3
repeat sub summing for for FR, SR, RR with Rsub channel as the target
- apply delays to time align main channels (aligning the l sub to the l speaker and the r sub to the r speaker and verifying that they sum OK by measurements)
- apply gain to bring levels back up to the correct level
2 things are missing from this process because it's not obvious what you want to do
- don't you want to spread the low pass of the C across both subs? you seem to be aiming it at the right sub at the moment which seems odd, if so then you want to cut that channel by another 6dB and then include it in the summing for each sub
- where is the LFE channel going? spread equally across both subs? if so treat as per the C (attenuate by a further 6dB and add to both subs)
Also why the different corner frequencies for the crossovers? Different speakers in each position?
A default starting position for the amount of attentuation to apply is
- reduce all main channels by 15dB
- reduce LFE by 5dB
- apply bass management
- raise all main channels back up by 15dB
- raise LFE channel up by 15dB (but if you do this digitally in jriver, you'll clip so IMV this is ideally done downstream either in an analogue gain stage or some other downstream piece (e.g. if your DAC has variable output levels and a mixer then you could crank it's output gain up and then trim the main channels in its mixer)
I think the above should work but I haven't worked through it in detail so someone might be along to correct me soon.
I recommend you write down the workflow you intend to apply and post it here for review... do a flow chart if you like