Currently, the ability to view and unban users are tied to the ban user permission, configurable via the api or the website.
This disallows groups from allowing a role to accept appeals without also permitting them to sabotage the group, in the event of malicious behavior.
Example:
The "Remove User from Group" permission is already separated - good.
However, if a user has the "Manage Group Bans" permission - they can bypass that by simply banning people from the group by username, which of course then removes them from the group.
This is both a vulnerability to the permissions system, as well as an inconsistency that makes it difficult to create effective management roles for the group.
I recommend splitting it into three permissions, and making "Ban from group" additionally require "Remove user from group".