Read the whole dialog box. The last two sentences are telling you how to proceed.
Something about your mkv file (encoding, sampling rate, number of channels) can't be digested by your TV, or if not the file itself then changes you have made in dsp studio are causing the problem.
You must adjust your settings to align with the capabilities of your output device. One example problem might be the mkv has Dolby true hd audio, you a configured to bitstream, and your tv doesn't decode Dolby true-hd; result=failure, and this exact error message.
If you can't figure it out, post screenshots of your jriver audio settings, dsp studio settings, and MC Audio Path (google it), and also describe how your pc connects to the tv (hdmi directly, through a receiver, what receiver, etc).