When I use the windows key or alt+tab to unfocus minecraft, the game volume is still at full, but it works as intended if I open a menu (such as pause menu or inventory) and click off the window.

Hm, I can't reproduce this. When I unfocus the game, be it through alt-tab or windows key, the volume is reduced as intended (tested by playing a music disc and opening to lan so it keeps playing when paused). On what MC version are you experiencing this? Multiplayer or singleplayer?

This seems to be fixed now, either from a new Windows update, 1.19.4, or something else