The B button on an Xbox Controller is basically always used as a universal back button, this is not the case in VRChat. It's currently only used the either close the Quick Menu or the Main Menu and doesn't even do anything in the Action Menu.
My suggestion would be to use it more like a back Button in all Menus, so the Quick Menu, Main Menu and Action Menu.
1. Main Menu:
  • Make it close popups like Avatar Details, Chose a List, Advanced Settings, etc.
  • Make it work like the back button, so when clicking on a World or User for example it will close it and go back to the page before that.
  • Make it close the Main Menu when nothing else it could close is open.
2. Quick Menu
  • Make it close the User Profile when opening it from the Here Tab.
  • Make it close the User Profile when opening it from the Friends Wing.
  • Make it close the View Photo popup you get when clicking on an imagine in the Notification Tab.
  • Make it close the Quick Menu when nothing else it could close is open.
3. Action Menu
  • Make it basically just work as pressing the Back button at the Top.
  • If on the Start Page of the Action Menu (Options, Emojis, Expressions) make it close the Action Menu