It doesn't feel like I'm getting xp from mining something.

It's basically done by which ores are valid for doubling in the config. You can also assign ore xp from there, but the array for it needs to follow the same sequence for the ores.