Hi,
When it's only for listening to audio files, it doesn't matter that much, just start the buffer low and fast latency, if the music doesn't skip, leave it like that.
If your music starts acting up, make the buffer higher and latency normal. Till the music sounds ok.
What audio card do you use in your HTPC? Does it match your high end audio gear? Do you use HDMI or the analog outputs of your PC audio card?