When I try to look up/down, the game thinks I'm trying to look left/right instead. When I try to look left/right, it doesn't work at all. I'm using generic USB controller on linux, version is controllable-fabric-1.20.1-0.20.2.jar

Seems like your controller does not have the correct mappings. If you go into the settings of Controllable (in game), go to the settings tab and click Download Mappings. Your game will be updated with the latest mappings which may resolve your issue. The only other option is to submit your mappings to https://github.com/mdqinc/SDL_GameControllerDB and it will eventually be included in a future Controllable uppdate.