You should implement the Fabric Item Transfer API and get rid of the hacky Inventory implementation that you have. The Transfer API tries to wrap Inventory's, but it assumes that the inventory essentially behaves like a chest or a furnace, which can lead to duplication/voiding issues if the inventory tries to do more things behind the scenes.

AppliedEnergistics/Applied-Energistics-2#5624

I'll get to it if I can, right now making sure the mod works at all is a higher priority.

#42 ties into this, but again, low priority.