The hover over tip says "Use ')' or ']' to group the rules". Is one operator an "AND" and the other an "OR"?
The tip actually says "Use '(' to -OR- Rules and '[' to -AND- rules". So it should be clear enough.
I will give you some more examples, in addition to what Marko linked.
[(RuleA
RuleB)
RuleC]
is equivalent to
(RuleA OR RuleB) AND RuleC
([RuleA
RuleB]
[RuleC
RuleD])
is equivalent to
(RuleA AND RuleB) OR (RuleC AND RuleD)
When you have nested brackets, such as one that begins with ([, you should make sure the the closing brackets are used in reverse order, i.e. ] before ) in this example. Example 2 in the post linked by Marko is an example of nested brackets that do not start with two brackets next to each other, like '[(', but it is still nested. In that example, '(' is before '[', and therefore ')' must be after ']'.