There is nothing that can be done in Options > Handhelds. Since the device is never correctly detected, there is no device in there to disable detection on. And the General > Features > iPhone Support checkbox makes no difference.
I would also love a way to disable detection attempts. It's pretty annoying, since the pop-up appears not just when my Nano is plugged in, but any time it syncs with iTunes. (Also, the pop-up's advice to "Please check for an updated version of Media Center" is a bit mean.)