Custom tools work quite well in anvils, but there are still some glitches that can cause players to be kicked out of the anvil in some situations. Luckily, they won't lose or get any items, and it doesn't happen so often. Nevertheless, it would be nice if this could be fixed. Unfortunately, it looks like the minecraft client is the bottleneck...