When trying to add a third enchantment to the terra blade with a vanilla anvil, the terra blade doesn't show the new enchantment that was added.

It seems to be irrelevant which enchantments are used.
Images
FML Log

All swords can only have one of the following enchantments: Sharpness, Smite, Bane of Arthropods.
It's not showing up because it isn't being added.

It does drain xp and consume the book though, or is that a vanilla thing

iirc you can't combine smite and sharpness on the same weapon.

Vanilla thing. If you have conflicting enchants one just overwrites the other