With "colornametags: true" all clients, that can see me in vanish-mode, crashs with this message:

"received string length longer than maximum allowed string lenger 17>16"

I think my MC-username with TagAPI color-code is too long for the clients and cut the name would help.

TagAPI trims usernames. You aren't using TagAPI. My bet is the hacky spigot dev builds. Downgrade to the recommended build until I release the next TagAPI update, then update to the latest.