You understand correctly, but just to be absolutely clear:
At my home entertainment center, I have a PC running JRiver MC. That PC has an AudioQuest Dragonfly DAC and on-board audio. A song is playing through my sound system using the Dragonfly. I go to another computer, log into the home entertainment PC via MS Remote Desktop. Immediately on connecting, the music stops. On the remote connection, I see the home entertainment PC screen, with MC open, but not playing. When I open the Sound/Speaker configuration (via Control Panel or the System Tray), the only Playback Device available is the on-board audio. The Dragonfly has disappeared. I go back to the home entertainment PC and log back in. The Dragonfly is available and I can restart and play the song.
I have used TeamViewer in the past, and re-tried it just now. It does work, but, it is unacceptably slow, clunky and complicated compared to MS Remote Desktop.
By the way, the described problem is the same using Plex as my media server. It is also the same if I try to use MS Remote Desktop App for iOS on my iPad. The problem appears to be a combination of MS Remote Desktop and the AudioQuest Dragonfly DAC.
I will be experimenting with using the on-board audio directly to see what happens.