GPU and FPS tanks down to 25% and 30 fps after about 5 minutes of playing the game, Im running on windows and using mc 1.21.4 with fabric, everything is up to date. I had multiple mods/shaders/and textures and couldn't figure out why this was happening so I did a clear uninstall of minecraft and upon reinstalling I played it on fabric with fabric API being the only thing in the mod folder. everything was working fine so I slowly downloaded my mods back and played for about 10 mins to see if I noticed anything, midnight controls was the only one giving me this GPU and FPS issue, so I played it with everything else on, my shaders, textures, and mods but not midnight, FPS and GPU were fine after 30 mins of gameplay, I then tried playing with everything off and experienced this issue after 5 mins of gameplay with midnight and API being the only thing in the mod menu leading me to believe its truly midnight causing this.

Does this issue occur specifically on 1.10.4 or also on prior versions?

Same issue for me. Also one note is that any type of mouse and keyboard inputs resets the FPS lag back to normal for some reason. If it starts lagging, just moving the mouse around fixes it temporarily, but you can also do this by opening a crafting menu or whatever and moving the cursor with either the D-Pad or the L stick

I just remembered that a feature was recently introduced into vanilla that reduces resource usage when no input is detected. Might be related to it.
I'll definitely look into this :)