This is standard behavior for every game I've ever played with a controller.

It's very frustrating not being able to exit menus with "B."

If it's broken, please tell me how to fix it. Otherwise, please add it when you can. Thanks.

bettermc4 all mods updated a few hours ago

You should be able to rebind "Close Inventory" to B. That simulates pressing the ESC key. I am not sure what version you are using, but if It's anything lower than the 1.20.1 version, you probably won't be able to change it.