I've quizzed dozens of high end users who, unless they sourced discs from the back floor of their cars, see no quantifiable differences.
Quantifiable differences are pretty easy to demonstrate. I just finished re-ripping about 2000 cds using dBPoweramp that I previously ripped using a non-secure ripping method (some itunes, some windows media player). The secure rips of those CDs produced different-sized lossless files with different checksums than the original rips in a significant number of cases. I would estimate about 10% or 12% had differences, often on discs that had no significant visible defects.
Additionally, there were also about 1% of the 2000 that couldn't produce a secure rip due to disc damage. Almost all of those had been ripped by non-secure rippers without complaint or even an error message. In at least a few cases, the discs in question had shipped to me already damaged, and they had only been removed from their cases once previously (to rip them).
The way dBpoweramp works, it rips once and if the checksum matches the accurate rip database it moves on. If the checksum doesn't match the database, it runs a second read and checks for differences between the first and second read. The neat thing about that is it alerts you when a disc might be damaged and where a non-secure ripper might not have gotten a good rip.
When I did comparisons of the new files from discs that needed a second pass by the secure ripping algorithm, versus the "old" files, in most cases I could clearly hear pops, skips, or other artifacts in the old files, especially on tracks that secure ripping couldn't get a clear read on.
Bottom line: secure ripping doesn't matter if your discs are perfect, and you won't hear any difference in those cases, but you don't know whether your discs are perfect unless you use some kind of secure ripping software, because visual inspection is no guarantee, and non-secure ripping software will not even tell you there's a problem.