Sometimes instead of retrieving 1 whole level it only gives 7xp which the amount for the first level only and other times it works fine. I thought changing the amount you retrieve was messing it up but it's kinda random.

Forge: 40.2.0
Reliquary Reincarnations: 2.0.19.1161

I guess this happens because it doesn't actually give you the xp but rather it spawns an xp orb on the player location or on the block you right click,

yeah this happens when you right click block instead of right clicking in the air. The reason that an orb spawns in that case is that you can actually use that to heal your mending items.