I don't think MC for Mac can work with an ISO of a DVD or BD unless they are unencrypted (which is not common). All commercially released Movies, TV shows, etc on DVD and BD are encrypted. MC for Windows has some hooks into some other subsystem of windows that allows it to play these as if they were physical discs and decrypt them on the fly.
You should be able to read an ISO of a DVD on your Mac by mounting it through Disk Utility. But I don't think that will help you to play it with MC. It would just show you the VOB files inside of the DVD image.
I'm relatively certain that HandBrake and MakeMKV can both read ISOs of DVD and BD. I'm also pretty sure that MakeMKV can unencrypt a DVD or BD and save the unencrypted video and audio into an MKV file at full resolution without any transcoding. So what you're left with is a video file that MC (and VLC and many other players) can play, is unencrypted, and has full audio and video resolution without any transformations that you might get from something like HandBrake.
Handbrake should be able to read an ISO of a DVD (not sure about BD) and save a compressed MP4 file with the audio and video contents in it. It will be a much smaller file, with a different CODEC (usually H.264), and possibly slightly lower audio or video sharpness. The file size should be much much smaller though.
In both cases, all of the menus are removed and you end up with the movie as a single file per title, which makes it much easier to use in my opinion. Navigating DVD and BD menus is always a pain. Sometimes pretty, but always slow and in my way.
Brian.