Issue only happens with two differently colored glass pieces connecting, not when next to solid blocks. Tried disabling and reenabling everything but it did not work.

The culling fix is only for the top and bottom sides of glass panes, not full glass blocks. The glass blocks not culling is intended behavior and works the same way in vanilla. The rendering errors you see are caused by CaffeineMC/sodium#38.

I think this is the sodium transluancy sorting problem. If it's that there is no fix possible currently except of removing sodium or waiting so that they solve the problem.