USB, Toslink, Coaxial Digital, and presumably AES all transmit a unified stream of digital audio. It's just audio and it's sequential and regular.
HDMI, on the other hand, does something completely whacky: Namely, it embeds the digtial audio stream(s) inside the video blanking intervals of the video frames! It's a very unusual and (in my opinion) convoluted way of transmitting audio. It means that the HDMI receiver has to disassemble each video frame, grab the relevant audio portion, then reassemble these audio pieces into some sort of coherent stream that can be further processed, decoded, played back, etc.
This disassembly and reassembly introduces the possibility of digital jitter. At this point, jitter is a well known, real effect, as opposed to being audiophile crockery.
Now, do you care about digital jitter? The answer depends on how serious you are about audio quality, how much of a concentrated listener you are, and how high resolution your audio equipment is. For casual listening, or even movie, TV, etc, you might not ever hear the consequences of digital jitter.
This is why I said you might care about it if you were "picky". Others could care less.
Brian.