That's the way Panel has always worked, there's really no caching of the loaded images. I would recommend switching to the list view in Panel as it loads images faster, but in the thumbnail view it can be pretty slow, yeah. Not really sure if it could cache images, especially if you're making constant changes to your library, as I could see scenarios where images could load for the wrong things and of course it could depend on the browser being used and its cache settings too, hmmm.
