I'm not sure I understand what duplicates you're trying to eliminate...
If you're trying to:
Find all of the duplicate files in your library, except those that are the files themselves in your playlists, to keep those playlists from breaking that's easy. You can do basically what MrC said (I think). But, I wouldn't. I'd search for those duplicates first, and then remove them (or keep the duplicate and fix the playlist to point to that file instead).
However, if you're trying to:
Find all of the duplicate files in your library, except for those files that are in the playlists, and any duplicates of those files in the playlists. This is weird, and I'm not sure why you're doing this, but you could:
Make a smartlist that finds all the duplicates of the particular type you're interested in.
Then, make a separate smartlist that finds all the files in the playlists, and their duplicates.
Finally, make a third smartlist that finds the first smartlist, and then NOT the second smartlist. This list would contain the ones completely "safe" to delete as you choose.
(Note: In this scenario you don't really need the first smartlist, as you could just search the Playlist folders directly as the first "search" in the third smartlist, but I'd do it this way as I'd find it less confusing and prone to error with checkboxes.)