I'm not exactly sure what the problem is. If someone could clarify, I might be able to provide a better answer. There are a couple of things I can point out.
When you create a Root Library Item at the same level as Music, Images, and Video (i.e. The main categories being Music, Images, Video, Root Library Item), you need to highlight the Root Library Item and use the 'Set rules for file display...' button. If you do not add any rules here, there won't be any items in the list. This doesn't apply if Root Library Item is nested in another group, but then it's pointless unless you're using it to further filter that category.
You might not want to add a Root Library Item unless you're creating a group at the same level as Music, Images, Video, etc. If you're creating a sub-category within one of these groups, it's probably easier just to add a Library Item.
Finally, when you're done making changes, you need to restart Media Center. You will not see any changes in JRemote, Gizmo, or Panel until Media Center restarts.