Let's start at the beginning. First, here's a link to the Wiki entry on sound setup in JRiver MC.
http://wiki.jriver.com/index.php/Audio_Setup#OSXThere's a lot of good info there; it's a good reference to look through.
But let's turn to your setup. Go to:
Tools > Options > Audio > Audio Device
What device do you have selected there? It should say the name of your DAC with "Core Audio" at the end. If that's not already selected, select the right device. Then, in that same window, click on Device Settings.
In the box that pops up, I recommend the following settings as a safe baseline:
Exclusive Access: Unchecked
Integer Mode: Unchecked
Channel Offset: 0
Buffering:
Software: 250 mS
Hardware: Hardware Default
*Most* of those settings should be the default, but double check them and change them if necessary.
When you're done with all of that, click OK on all of the dialog boxes until you get back to MC. Then quit MC and restart it. Now test audio again. Try a simple file like a CD rip (44.1kHz) or something. We can move on to high rez audio or DSD later. Let's get the basics working first.
Let me/us know how you do, and good luck!
Brian.