The relevant settings appear to be General - Behaviour - Double-click and Play button. I suspect you have the first set to Replace Playing Now (single) and the second to Play visible files. You would then get the result you describe if you are double-clicking to start an audio track, but using Play to start a video. Otherwise, the behaviour may be confusing, but is seems to be consistent.
I set Double-click to Replace Playing Now (all) and Play button to Play visible files because I want all visible audio tracks to be sent to PN when I double-click one track. Unfortunately, the same applies to video, where I want only the selected item to play. But I seldom notice that, as I tend to stop a video when it's finished, and then PN is cleared or replaced. I do agree it would make sense for there to be separate settings for video, and a Play selected option for the Play button.