DSD Bitstreaming takes a DSD input, bypasses all internal processing (except possibly packing it into a DoP stream) and sends it unmodified to the DAC.
DSD Encoding converts any input to DSD. If your input is a DSD file, it will be converted to PCM first, and then encoded to DSD again.
This intermediate PCM stage takes place so that you can apply processing to the signal (e.g. Volume Leveling or EQ) before encoding it to DSD.
I believe Bitstreaming takes priority over Encoding, so enabling both options will convert all PCM formats to DSD, but bitstream DSD inputs. (bypassing the PCM conversion stage)