So, the controlify developer (isXander) has implemented support for sdl in PojavLauncher, which makes PojavLauncher support controllers/gamepads natively via controlify and uses special libraries for it to work (file is attached). It seems that legacy4j uses isXander's libraries as well, however, the version number is different (3.1.3), which means, that the sdl libraries that legacy4j uses are older, and does not contain all the patches, which isXander made, to make the gamepad to work properly on PojavLauncher.
So, is this possible for you to update these libraries to this version (3.1.9)?

Thanks in advance.

libsdl4j-natives-3.1.9.075c033-linux-aarch64.zip

Added in 1.8.0.2517.0, thanks for reporting