Help me understand this as the change in jriver occurs before its signal got sent to my AVR, so the boost on my speakers assuming the subs also got boosted in jriver (which is another question: why the subs got boosted if LS filter was applied to speakers only, the sub was not checked in this LS filter) will be crossovered by my AVR setting: small/120hz no? Thanks.
Yes, the change occurs before the signal goes to your AVR. That's the problem. The AVR can only re-reoute what you send to it, and if you modify the left and right speaker before sending it, any modifications will get re-routed too. Adding boost to the speakers before bass management will always boost the sub whether you select it or not, because you're boosting the material before you feed it to the AVR.
A shelf filter stops rising at a certain frequency, but all frequencies below it are also boosted. So your shelf may stop rising around 120Hz (depending on the Q), but all frequencies from 120Hz on down to 0 are still boosted +6dB. So when your AVR re-routes the bass from your speakers, it's re-routing the boosted bass, which means the bass from your speakers in your sub will be +6dB louder than it used to be, and +6dB louder than the native LFE content.
Imagine that the normal volume of the bass you redirect from your speakers at 100Hz is 30dB, which gets redirected by your AVR to the Sub and added to the LFE. You've added 6dB of boost to material at 100Hz and below, which means the redirected bass from your speaker is now 36dB when it gets added to the LFE. That will not only make your sub louder, it will change the relative composition (speaker bass will sound louder than LFE bass, etc.).
If you want to do bass management as your last step, there's no way to do what you're doing that will work the way you want. You need to do bass management in JRiver before the shelf to have full control over this kind of thing.