Thanks for the replies. I understand more now.
I'm not sure why you'd set the output to 7.1 at all if you only have six speakers unless you're doing active crossovers or something?)
Another 5.1 setup I have, I use 6 speakers plus sub. I use 8 channels, combine the rear and sides, then use one of the rear outputs for a second center channel speaker. I was thinking of keeping 8 channels to have those ports available for other tasks. I will do a separate post later, as things will get a little fancy with this setup as the dac allows for 8 channels of analog input too. This is actually easy to deal with though because the numbered input simply passes through to the same numbered output. The dac outputs single ended and balanced simultaneously so I can use my balanced amps with unbalanced sub.
Regardless of how the channels are ordered in different scenarios, there is consistency to it and it looks like I can simple set output to 5.1 or keep at 7.1 and do some channel mixing in parametric equalizer.
More info about my setup:
I did have JRSS mixing on. I see now there are options for No Upmixing, and Clone Side Channels to Rear. The latter may apply to my situation.
I am connecting to DAC via USB. For linux there is no DAC software or dsp applied in the DAC. I agree the channel order is weird. This is what led me down the path of confusion. Testing one channel at a time to see what output is what. If I had to guess, the channels are weird because the DAC is designed to be used with minidsp software on Windows and Mac. They probably reroute it in their software to make more sense. From what I have seen, their software has some similar functionality to MC DSP, which would make it redundant for someone who only uses MC. I have not used their software other than for firmware updates.
The DAC does accept earc hdmi, but only LPCM. I plan on giving this a go soon, when time permits. It will be the first time I will try using hdmi for audio. This is dangerously close to using a receiver, heaven help me. I am antireceiver, or a receiverphobe, however you want to label me, I am fine with either. minidsp documentation says there is different channel order when using hdmi, so I will see how this goes.
I will do some more tests, and hopefully be less confused. I will post again if something truly doesn't make sense, but either way I will have working audio because my setup does currently work in 5.1.