apologies to the OP but I think a variable Q high/low pass filter is a pretty niche thing to add, particularly if it were the only change you made. It would benefit a pretty small number of users.
I/we created this thread years ago on the subject of changes to dsp studio -
https://yabb.jriver.com/interact/index.php/topic,99096.msg685880.html#msg685880 - and I think all of those are still valid feature requests.
IMO the 2 most valuable/useful things you could do are
1) add a new filter type "advanced biquad" to PEQ
2) provide some easier, less labour intensive, way to add/remove PEQ filters (e.g. to paste in a batch of filters in a text format)
With the first one, you give the user the ability to implement pretty much anything they want including any order of Butterworth or Linkwitz-Riley high or low pass filter (as you can simply create those by stacking biquads)
With the second one, you may it feasible to let people easily use external filter design software and import that directly into jriver quickly and easily.
Even better if you make point 2 available via MCWS so it can be automated easily. If you did this then I could integrate it with an app I wrote (
https://beqdesigner.readthedocs.io/en/latest/) which is used over at avs (
https://www.avsforum.com/forum/113-subwoofers-bass-transducers/2995212-bass-eq-filtered-movies.html). atm jriver is widely recommended as the way to do this on a computer but it is pretty tedious and error prone to do (because of the manual data entry and because of the way jriver uses S as the parameter for a shelf filter)
FWIW I rarely use PEQ in jriver because it is so tedious and time consuming to use if you want to add a lot of filters.
On the other hand, if you just want to add this particular feature and need the biquad formula for it then you can refer to
https://github.com/3ll3d00d/beqdesigner/blob/master/src/main/python/model/iir.py#L555 and
https://github.com/3ll3d00d/beqdesigner/blob/master/src/main/python/model/iir.py#L607 for a working implementation of how to calculate the biquad coefficients.