The Group By feature not only defines the grouping and header text, it also defines the sorting for the file list, which is essentially ASCII sorting. So if you want to sort the groups, you need to sort by a custom user field that includes the correct output to allow ASCII sorting to work in your favor. An example would be [Year] - [Album].
The stock "Album" Group By setting is special - it includes a right justified Year on the right portion of the header, which is used for sorting, but displays Album on the left. And "Album (by date)" does not show the Year, but does sort by the album's date.
The Set rules for file display sorting sorts, for example, the values present in a pane's column. Or the order of the presented categories in a Categories view. Note that Standard View includes four sub-types: Panes, Categories, 3D and File List views.