(My apologies to those who don't like reading anything lengthy. This is a long post, in response to an important topic.)
Ok, since you ask...
First, I would say that some of the things you describe are not support. Support is directly addressing a user issue in a way that provides value to the user, and the costs directly associated with that. So for example the costs of hosting the forums are support, but the costs of hosting the JRiver website are not. The costs for purchase payment processing are not support. Similarly, directly answering user questions on the forum is support, but the entirety of time spent reading the forums is not; here's why: a smart developer or product manager must monitor and take into account user feedback. Seeing (reading) what questions and problems are raised on the forum are essential to being in touch with the needs of your user base and future product development and refinement, and the time spent doing that is analysis and research, not support. Failure to do it means a product disconnected from the needs of the users. In other words, if you read 100 forum posts and answer 1, you count the time on the 1 as support, not the 99. So some of those things are merely the cost of doing business, not the cost of support.
But frankly, I think that's minutia and beside the point. Whether your support costs are 50k or 100k, although obviously the difference matters to you, is not the central issue and I don't want to argue about it.
We've also previously established that you do not believe that better documentation reduces support costs and thus increases profits, so I will not beat any further the dead horse of providing more comprehensive documentation as an initiative. That is also not the central issue.
The central issue is the appropriateness of charging for the forums.
If JRiver wants to offer a paid support option, where JRiver developers or support engineers would actually be obligated to respond in a timely and complete manner to a question or case, that would be perfectly reasonable, perhaps even marketable. Lots of companies, including Microsoft and Cisco, use this approach. That is one option.
Or maybe you could adopt the Uber model: you charge for a brokerage service. When Roderick answers a question, the user is charged $10, and you keep $3 and Roderick gets $7. If you think about it, you can probably see the reason Uber wouldn't stay in business if they kept the whole $10. That's essentially what you're positing.
But charging for access to a user-to-user forum... I think that this is an odious idea. On several levels.
Charging for the forums is tantamount to turning MC into a subscription: since it does not come with good quality documentation, being able to ask questions on the forums is essential to many users, especially those less technically adept. For these people it's simple: Sure you can get the software for the regular purchase price, but don't expect to be able to use it fully or effectively unless you pay for the forums too. It is only access to in-depth interactive support that makes the software, which is difficult to learn for many people, a manageable solution.
Further, 90% of the questions that are answered on this forum are answered by users, not by JRiver staff. People like Marko, Brian, Roderick and others (even me, in my small way) spend a lot of time answering questions and writing detailed tutorials for no other reason than they want to help other people. JRiver doesn't do this, the users do.
You've said JRiver couldn't answer all the "customization" questions due to the number of permutations, even if you had a full time paid person to do it. JRiver gets around that problem now by just ignoring questions they don't want or have time to answer, and leaves it to the users. That's fine, but that's not how a legitimate paid support model works. If you want to be paid for support, you have to provide support.
Charging for the forums, as they are currently operated is nothing less than making money of someone else's work. Charging for access to the work of others, when they are donating it out of a sense of charity, and donating not to JRiver but to the community, is not only repugnant, it's offensive.
Having a business model that depends nearly entirely on unpaid charitable acts to provide direct user-to-user support walks a fine line. Charging money for it crosses the line, into exploitation. Even Microsoft, experts at extracting every last nickle, do not charge for access to their community forums.
There was mention about the support costs associated with old versions. How often do JRiver staff actually support old versions? It's the users that answer those questions.
When someone like Marko is donating his time and wants to answer a question, to say that a user can't ask Marko a question about MC23, and thus Marko can't answer it, unless you are paid is simply beyond the pale.
A big part of the reason why JRiver has done as well as it has, is because it has a vibrant, and charitably minded, user community. It would be unwise to stifle that, or try to be overly controlling of it.
It's also bad business, because the forums are fungible...
The price elasticity of a good is constrained when that good has acceptable or perfect substitutes. Many would argue the the MC software itself does not have good substitutes, because of its feature set. Someone who needs the capability of MC views will not switch to WinAmp, because WinAmp just can't do what they need... On the other hand, the forum has many perfect substitutes. There are many, many, many audio forums on the internet, where JRiver users could move and setup a set of boards to discuss this software, and on those forums they could have just as good access to other JRiver users as here. The only thing that distinguishes this site from any other is access to the JRiver developers: if that access goes behind a paywall, then the entire user base can move to another forum with zero loss. That is not something JRiver can control. Charging for these forums will simply encourage the community to move elsewhere, to a different forum where they can discuss the software freely. And it won't only be the people who ask questions that move, it will be the people providing the content, the answers.
There are other better options. The simplest: If you don't think you're making enough money off MC, charge more for it.