Seems like we cant remove the mod at all when removing the jar, is there another way to remove it?

Also the key doesnt work on 1.20.1

If it still doesn't work, can you explain in details what you do, and what Minecraft version and mod version you are using ?

The mod should be removable by simply removing the JAR. You can also clean the jobs folder in the world or server folder. The steps to find this folder are explained in the Wiki.

Also the key doesnt work on 1.20.1

Can you create another issue for this and explain if "J" doesn't work, if you see the key binding in the settings and if changing the key binding solves the issue ?