There isn't currently a good "default" way to do what you all are describing. All channel counts above 7.1 have a hard coded mixing target of 7.1. There have been feature requests in the past (which I've signed onto) to be able to specify the channel output and mixing targets separately, but that hasn't gotten a lot of traction.
But both of you can do what you want, just not solely with JRSS. OP, you just need to manually downmix from 7.1 to 5.1 using parametric EQ. Davewantsmoore, you'll need to manually downmix to stereo, unfortunately. If you had four-ways instead of five-ways you could use the "2-channel inside a 7.1 channel container" option, but there is no equivalent option for five ways at the moment. The JRiver team is generally responsive to user needs, but they've also got a lot going on development wise, and this is, unfortunately, kind of a niche area.
I run bi and triamped speakers in various applications myself, so I feel your pain, and being able to specify the mixing target more granularly would be handy for me too, but in the meanwhile we can accomplish what we want manually using PEQ.