I modified launcher script and something else to make HMCL, the launcher i use( https://github.com/HMCL-dev/HMCL ), to support launching gtnh 2.5.1 with java 17.
however, the relauncher window, which is supposed not appearing, appeared here and override the launcher config from previous launcher.

herez my HMCL config and launcher script:
GTNH-2.5.1.zip

client setup before upgrading lwjgl3ify 1.5.16 -> 2.0.0:
config.txt

i dont think relauncher window should appear if im using java 17, since i handled args and java previously, it should only appear while on jre 8, no matter the launcher i use.
or we should add a config tweak to disable it manually if relauncher is unnecessay for such situation

The trigger for the relauncher is not modern Java, but whether RFB has booted the game, as lwjgl3ify relies on it for its key asm transformers. It does so by checking a launch blackboard value that's set early in RFB's init process, so if you launch it with java 17 using some other method it will not be there, and lwjgl3ify 2.0.0 would most likely not work anyway.
You're probably missing the main class override to rfb's main, or similar.