Ben, are these m2ts files simple rips from BDs? If so, you need to know which video and audio codecs are on the original disks. Not all BluRays are h264 compressed - a large number use VC-1. In those cases coreavc will not work to decode the video.
Also, the audio might fail unless the discs contain DD or DTS tracks. Unless you have ffdshow set up right (and a recent version at that), you will not be able to decode TrueHD or DTS-MA audio. You may not even be able to extract the core lossy tracks. AC3 fiilter will not help if your setup cannot even recognise the audio format.
Here'swhat I would do in order:
1) Find out which codecs are used. Windows Explorer can do this if you choose codec as one of the headings you can sort the containing folder by. MC can also do this when you import the mt2s, I think, in recent builds.
2) Assuming it's AVC/h264: Rename the m2ts extension to mkv. Now will it play in MC? (This test assumes you you can play back "real" mkv files in your system). If it plays, then set up the m2ts playback the same way you have for mkv.
3) Download and install MPC-HC. Will that play the m2ts file? If it does "out of the box", you can set up MC to launch MPC-MC for m2ts file playback. Or... you can register the codecs being used by MC-HC as standalone codecs for other directshow players, and then try to set up the J River engine to use them. This is what I have one. You may need matroska splitter (Gabest's/MPC-HC's internal version), and the mpc-hc audio renderer too.
In particular, you will have to read about setting up ffdshow to decode or bitstream TrueHD and DTS-MA, or to at least understand these formats so that SPDIF can pass through the lossy cores of these formats.