Take a look at the Noire skin.
Maximize.png, Minimize.png and Close.png are examples. When checking if the mouse is over the control we check it against that last image. The upshot is that if you have round buttons you can actually end up with round hit boxes. (Hit circles, I guess.)