You should implement the Fabric Item Transfer API and get rid of the hacky Inventory implementation that you have. The Transfer API tries to wrap Inventory's, but it assumes that the inventory essentially behaves like a chest or a furnace, which can lead to duplication/voiding issues if the inventory tries to do more things behind the scenes.

AppliedEnergistics/Applied-Energistics-2#5624

That should be quite easy thanks to SingleVariantStorage<ItemVariant> though. 😉

It is indeed quite easy -- in fact, I did it on an experimental branch recently.

Considering that the current Inventory implementation has caused plenty of breakage over time, I'll see what it takes to backport to 1.x.