Also note that according to Sony your TV can play MKV files, with DTS Core audio, and a bunch of other formats.
https://www.sony.com/electronics/support/televisions-projectors-lcd-tvs/kdl-47w802aOf course, it depends on what video format is inside those MKV files, as MKV is just a container, not a format.
As 2 Channel, 5 Channel Stereo, DTS Neo 6 all work, and all are 2 channel input formats, it sounds like something can't or isn't handling the 5.1 channels in DTS. But I don't really understand how you have everything connected up. It sounds like you are sending the video and audio via Airplay to the TV, and then sending the audio back to the receiver via the Audio Return Channel (DIGITAL AUDIO OUT (OPTICAL). That is a lot of pieces in the chain, and each might be causing the issue. In fact I wouldn't be surprised in the TV is set to output 2 channel audio via the ARC, and then the Denon receiver is able to play those in 2 Channel, 5 Channel Stereo, DTS Neo 6. There may be a setting in the TV to pass DTS 5.1 back to the receiver.
As an optical cable can handle the 5.1 channels in an encoded DTS signal, connecting from the laptop directly to the receiver using an optical cable would make sense.
But you might want to check what is actually in the MKV files you are creating using MediaInfo, as Brorsoft doesn't list DTS or DTS Core, only listing "DTS (remux)" in its output formats, which could mean anything. See
http://www.brorsoft.cn/blu-ray-ripper/reference.htmlCan't AnyDVD HD rip the Blu-rays for you, rather than using Brorsoft? I use DVDfab and it certainly can rip Blu-rays. Unless you are using Brorsoft to selectively save specific audio tracks and convert to specific video formats, and to burn the result to a disc. I just rip the whole Blu-ray structure and use that, but I have disk space for that, and I guess your laptop doesn't.