It would be nice to be able to disable the Ore Magnet in configuration, since its functionality is a bit game breaking (replacing blocks as it pulls ores towards you, potential griefing of homes that keep ores nearby).

To note, the blocks it replaces are a small whitelist
https://github.com/TeamTwilight/twilightforest/blob/1.15.x/src/main/java/twilightforest/item/ItemTFOreMagnet.java#L208-L211

Ore Magnets are loot items that cannot be crafted and thus can be removed from loot table chests via a datapack. If that doesn't help, you could also configure the Tags it uses to be completely useless.