OK, I played around with this a little and here's what I've come up with:
I think that looks almost tasteful (notice how I reduced the icon size) but if you want to create your own icons, here's how I did it:
1. Choose icon online.
2. Bring it up in a picture editor. Use the magic wand to select the icon, so separating it from its transparent background.
3. Invert the selection so the transparent background is now selected.
4. Flood fill the transparent background with JRemote grey. The RGB colour values to enter in your colour picker are 230 230 230 E6 E6 E6.
5. Resize image to 512 x 512 pixels. I'm not sure if this is necessary.
6. If your icon comes up too big, open it up in the picture editor again and keep adding grey borders around it and resizing the total image to 512 x 512 until you've squeezed your icon down to the desired size.
7. Follow marko's instructions for installing the icon.
Many thanks indeed to marko for coming up with this idea. I think it really improves the user experience.
Here's a link that you can use to download my icon in the event that anyone wants to try it:
https://www.dropbox.com/s/4z9medfnjdr01kq/JRemote%20icon.jpg?dl=0