I've found two separate conditions that cause it to freeze.
1. One of my rules accidentally included the zone it was switching to in the list of zones to stop playback in. It doesn't stop playback but causes it to freeze for 20s. (prebuffer)
2. If I've got a smartlist that contains 2ch and 6ch music, it freezes if it has to switch zones during active playback.
#2 is a bit complicated to explain.
2ch files go to my DAC, 6ch files to to my AVR.
I have a smartlist which plays random albums so it might be a 2ch or a 6ch album.
If all playback is stopped and a 2ch or a 6ch album plays, playback starts immediately.
If I hit the random smartlist again and it's another 2ch or 6ch album, it plays immediately.
If it was playing a 2ch album and the next album is 6ch, it freezes for 20s. (prebuffer)
There is a separate issue where I actually don't want my smartlist to include 6ch albums at all, and can't figure out how to prevent that, but that's for another topic. (not as easy as it might sound)