Your phrase "regex foo" had me scratching my head. What does that mean in plain English? I'm a Brit, so some American slang goes over my head
Several years ago, I made some very limited use of regex code in my website design activities, but never gained any meaningful skills. Such knowledge as I had back then has now been almost totally forgotten, so I'm unable to offer any real help with your regex expression. That said, I did confirm your regex101 test results. Your basic expression does indeed highlight (in pink) any suffix wrapped in round or square brackets.
That suggests you wanted your new calculated field [Album Title] to only contain such suffixes when found at the end of an Album title, in which case should your new field not be called [Album Suffix]?
I created a new calculated library field called [MyRegexTest] using your full expression
Regex([Album], /#^(.+?)((\s*\[[^]]*\])|(\s*\([^)]*\)))?$#/, 1). Unlike you, this generated a result for all my audio files, but not the result I was expecting.
For my non-classical discs, the value of MyRegexTest was [Album] by [Artist], as in the following example.
Greatest Hits (2011 digital remaster) by
Queen.
Note that there was no extraction of the text in round brackets. For my classical discs, MyRegexTest was always set to [Composer].
Things are getting curiouser and curiouser...