Camera shouldn't save its previous state in the gravity direction but instead pick the closest possible one in the current gravity direction and apply it after gravity changing

What do you mean? Currently it keeps your world looking direction the same when changing gravity, are you saying that it would be better if it instead kept player relative looking direction the same?

Yes, this is what I meant

Added a config option to disable it