Just looked around the airplay site and came across this
WASAPI Exclusive mode is the correct setting. That article is just providing information on how to hear sounds from other applications or system sounds while MC is playing audio. Option 3) is best.
Also, that article doesn't inspire confidence. It refers to WASAPI as "relatively recent". It was introduced with Vista, 10 years ago!
A better solution, if there are local PC speakers available, is just to set the local speakers as the Windows default audio device, and set the Airfoil as the output audio device in MC. Although it looks like maybe Airfoil only captures the Windows default audio device, so it may not be possible to do that. I would still use WASAPI Exclusive mode in MC for the best sound.
What you need to do is find out what audio formats Airfoil supports. Bit-depth and Sample rate. I couldn't see that anywhere on their site. Do the M4A files that won't play have higher Bit-depth and Sample rate?
You aren't trying to Bitstream to Airfoil I assume? Bitstreaming is off?