I play fantasy minecraft which has mariums soulslike weaponry on a server. I had a sowrd of freyr with the shaped glass enchantment on it. when i tried to combine it with mending on an anvil, it suddenly shattered like glass and was gone

Shattered like glass?
So it basically despawned when you applied mending or..?

Yep, it made a Sound like shattering glass and despawned after it. I thouhgt it had something to do with the shaped Glass enchantment, but i recreated this situation in a singleplayer world and it worked

I don't really know how that enchant works, but I am guessing the issue has something to do with that mod since I am having no problems adding enchants myself. Maybe make an issue post over there for that dev.