If using Modern KeyBinding, movement keys won't work inside inventories.
This is because Modern KeyBinding ports key conflict features from Forge to Fabric, which adds additional checks that disable the keys while inside inventories.
InvMove has a workaround for (Neo)Forge, but porting the workaround to Modern KeyBinding on Fabric will require explicit support in InvMoveCompats.

Until then, the only known workaround is to remove Modern KeyBinding.