Fix this by adding KeyHolder "scopes" (Inventory, Game, All, etc..), so the keyholder is active in the specific scope.