Asked/searched a few places around and here's what they told

• Fabric Loader/API - concerned about breaking vanilla/not in scope
• Sodium - could fix but not interested
• Minecraft bug tracker - MC-265322 and it was supposed to be fixed in 1.20
• No Prying Eyes - a small mod where I thought this fix would be interesting to implement in, no response yet