People who don't have the vanish permissions were vanishing on my server. I don't know what the problem is.
Im using GroupManager as my permissions manager.

Ask the GroupManager folks for help with your permissions setup, as anyone able to /vanish has the vanish.vanish node.