Add a connector block to the bottom of the factory block.
Add a power/inventory block to the bottom of the connector block.

This will be a proxy to the main factory block and give you access to 5 sides.
Any of which will accept power.
Any of which will be used as an inventory output.

This will mean that you wont need the inventory in front of the factory.

bd5913c