Hi Matt,
I have, as yet, no opinion on the matter, as I haven't investigated it myself. It's worth noting that in the 1970s a bunch of horrible sounding solid-state gear was sold, based on very low distortion with steady-state tones. Later, the phenomenon of TIM (transient intermodulation) distortion was discovered and explained why this gear measured great but sounded bad -- to those who really listened -- with music. Since then, audio manufacturers have reduced the amount of negative feedback used, especially global feedback, and many, probably most, listeners and critics approve.
The explanation I have heard for CPU-loading effects is that heavier CPU load is somehow jitter inducing. Why this should make a difference with modern DACs, which usually have methods to defeat jitter, I don't know. However, we must remember that the history of audio, as well as being littered with green pens and magic pucks, is also marked by statements by golden ears that something wasn't right -- though it measured fine -- and later confirmation from the engineering community. Indeed, my first CD player was just terrible sounding, though critics loved it. The gear I have now, almost 30 yr later, is tremendously more engaging. That was achieved through engineering, not magic, once the engineers figured out what the problem was and how to measure it.
I do think you are measuring the wrong thing in this case, but I don't think we yet know what to measure to quantify this effect. It may turn out that nothing measure it, as it's some kind of mass delusion; or it may be some kind of jitter modulation that is not well removed by the methods now in use. Or something else! It's great that you are trying to figure this out. You're not there yet!