I haven't used HDMI with MC for Mac yet. But it should work like the windows version. By default, MC is going to decode any formats you give it: AC3, DTS, and almost anything else. It gets decoded to uncompressed PCM. So your 5.1 channel AC3 track turns into a 5.1 channel PCM track. MC should then send that down the HDMI connection to your receiver. Most receivers can handle multi-channel PCM, so you really don't need to do anything. It should "just work".
However, if you prefer to have the receiver do the decoding, you can do that too. To tell MC to not decode certain formats, use the bitstreaming option:
Tools > Options > Audio > Settings > Bitstreaming > Custom > (select your formats)
Note that bitstreaming means that MC can NOT touch the bits that get sent at all. So, no volume control, no eq, no room correction, etc. Nothing. Your receiver will need to do any of those functions.
As I understand it, the "best practice" with MC and movie audio is to have MC decode it to PCM, so it can do any additional processing you might want. But you have both options: MC doing decoding, or your receiver doing decoding.
Brian.