It would be great to have a custom compacting rule like in storage drawers config that doesn't need crafting and uncrafting recipe to be able to compact inside compacting drawer.

For example ex nihilo's 4 ore pieces compacted into its raw ore, even tho the raw ore can't be uncrafted back to ore pieces.

And what would happen if you tried to put a Raw Ore into the drawer? Would the number of Iron Pieces go up?

It is not a compacting recipe. The storage drawer isn't counting 2-3 types of items, it is holding 1 thing (the smallest unit) represented in 3 ways. So it only makes sense for bidirectional conversion.

If you are playing a pack with Ex Machinis (ATM7 To the Sky?), then that mod does provides a Flux Compactor block specifically for crafting all of the Ex Nihilo Pieces into their respective raw ores

Yes, the number of ore pieces would go up,

It might break some balances, but at least mod pack devs have configuration options to put it in the pack or not