I haven't had a chance to test if this bug also happens in outside of the Infinity mod pack(v1.3.1+).

I added the phantom bridge sigil to a bound chestplate, and it seems to create the spectral blocks 1 block to high(instead of below the feet, it creates them at the waist) and I'm assuming that because they're made one block to high that would be why they don't despawn.

Yes, this is the reason why they don't despawn.
The problem is that on the client side, the player's eye height is accounted into his position, so their position appears about 1.6 blocks higher than on the server.

This issue has been worked around in the case where you use the sigil normally, but not when it is in your armor.