Far from ideal, but the only two ways I see to get this are by either creating the rule:
playlists - is not any - ...
and then select every playlist in the list.
click OK, then open the playlists action window to verify that nothing in the list belongs to any playlist.
The list of playlists presented by the smart list wizard includes smartlists too. Be careful not to select any smartlists as you would undoubtedly end up excluding valid files if you accidentally selected one.
If you have a massive list of playlists, you could work through them more quickly by using the 'down' cursor key to move the selection to the next playlist and hit the space bar to 'tick' it... [down],[space],[down],[space],[down],[space] etc. etc..
OR
That's a long way to go for a shortcut, and the only other way I can think of to reach your target is to ensure all your playlists are in a playlist group with no smartlists.
Select the group, and all the files in that playlist group will be listed.
Apply a tag to all of those files. Either use an existing, unused tag, or create a new one specially, then create a smartlist rule that excludes all files containing that tag. The result should be all unplaylisted files.
Messy... Good Luck