Agreed, a smartlist is the best solution. Just determine (or create) a field or file value that can be included or excluded as desired. It could be the file name as suggested.
Or, to easily handle files of any name, you could put a certain value in a particular field that identifies such tracks as excluded from your "music only" playlist. The Genre field is an obvious place to specific that a given track is a Test or Sound Effect or anything else that you want the smartlist to exclude. Even the Rating/Stars system could be used, exclude 1-Star tracks.
I use the Keywords field for each track to identify it in various ways, then exclude certain Keywords from my general music listening smartlists. My wife doesn't want to hear my tracks of test tones, random noises, 1901 historic recordings, talking and comedy cuts, christmas songs (those are included in a different smartlist) and other annoying tracks.
It can be a simple one-time task. Consider that a smartlist can include AND/OR exclude. so instead of having to tag all the songs you want to hear (though this is easy), you could just tag the tracks you do NOT want to hear. Then use the smartlist to include or exclude as desired.