A key thing to understand Hawi is that DLNA is a file-based protocol. So MC doesn't really play anything. It sends files to the DLNA Renderer (your Marantz). If you don't use DSP Studio, MC sends the file as-is. If you DO use DSP Studio, then MC transcodes the original file to whatever you specify, then sends the resultant file.
In either case, it's up to the renderer (your Marantz) to understand and play the file. If it doesn't understand the file, end of story. You either convert it to a different format, or you don't play it.
Even if the renderer does understand the file, MC doesn't really know what the renderer does with it. MC can send a 6 channel file, the renderer can say "ok thanks" and then still play the file as 2-channel.
If Marantz support can't tell you what formats it supports via DLNA, then you really only have 3 options:
1. Google
2. Whitebear DLNA Analyzer (limited precision)
3. Trial and error, trying progressively increasing channel counts, sample rates, and formats, until you find the limits.
Here's a report for the Marantz SR-7008, which should be similar to yours. DSF is not listed. It indicates LPCM supports a maximum of 2 channels.
Declared Supported Audio, Image & Video Formats
===============================================
http-get:*:audio/L16;rate=44100;channels=1:DLNA.ORG_PN=LPCM
http-get:*:audio/L16;rate=44100;channels=2:DLNA.ORG_PN=LPCM
http-get:*:audio/L16;rate=48000;channels=1:DLNA.ORG_PN=LPCM
http-get:*:audio/L16;rate=48000;channels=2:DLNA.ORG_PN=LPCM
http-get:*:audio/mpeg:DLNA.ORG_PN=MP3
http-get:*:audio/x-ms-wma:DLNA.ORG_PN=WMABASE
http-get:*:audio/x-ms-wma:DLNA.ORG_PN=WMAFULL
http-get:*:audio/mp4:DLNA.ORG_PN=AAC_ISO
http-get:*:audio/3gpp:DLNA.ORG_PN=AAC_ISO
http-wavetunes:*:audio/x-ms-wma:*
http-get:*:audio/wav:*
http-get:*:audio/x-wav:*
http-get:*:audio/flac:*
http-get:*:audio/x-flac:*
http-get:*:audio/x-m4a:*
http-get:*:audio/x-mp4:*
http-get:*:image/jpeg:*
Evaluated Supported Audio Formats
=================================
audio/L16 LPCM Dlna 44100 1 16 Yes
audio/L16 LPCM Dlna 44100 2 16 Yes
audio/L16 LPCM Dlna 48000 1 16 Yes
audio/L16 LPCM Dlna 48000 2 16 Yes
audio/mpeg MP3 Dlna * 2 * No
audio/x-ms-wma WMABASE Dlna * 2 * No
audio/x-ms-wma WMAFULL Dlna * 2 * No
audio/mp4 AAC_ISO Dlna * 2 * No
audio/3gpp AAC_ISO Dlna * 2 * No
audio/wav WAV Fuzzy * * * Yes
audio/x-wav WAV Fuzzy * * * Yes
audio/flac FLAC Fuzzy * * * Yes
audio/x-flac FLAC Fuzzy * * * Yes
audio/x-m4a AAC_ISO Fuzzy * 2 * No
audio/x-mp4 AAC_ISO Fuzzy * 2 * No