That's a fair enough "by the ear" comparison.
I assume you are trying to tell if you can hear a difference between an MP3 and a FLAC file to decide how to store your library? Or something along those lines?
Foobar has a pretty cool add in you can use to really gauge the differences for yourself, without injecting listening bias.
Since Foobar could maybe be seen as a competitor, I won't link to their stuff directly, but you need the install of Foobar, and then you want to do an Internet search for Foobar ABX Comparator.
It asks for the two files, and then will let you switch between the two dynamically, and randomly, playing back from the same location in each file. No stopping, starting, switching, restarting, switching, and adding the bias of "but it's lossless, it MUST be better" into the mix.