At the moment there is no way for any mod to have custom shaders with iris installed.
I do understand why compat on irises side is practicly impossible,
but i also dont understand why there is no way for mods to implement compatiblity.

Describe the solution you'd like
A way for a mod to register a Shader / a replacement shader for its own shader as compatible with iris so iris does allow it to render. This way the burden of compatiblity is shifted to the mod itself.

At least with sodium there's always the same set of shaders, but when iris loads a shader pack the shaders are whatever the shader pack says. There's no way to combine the shader from a shader pack and the shader from a resource pack if they're trying to render the same thing.

Im not saying Iris needs to find a way to make a mods Core Shaders work correctly.
What im requesting is that iris at least implements a way for Mods to use Shaders in some capacity.
If that needs to follow strict rules about vertex formats with only limited customdata and have heavy restrictions on what is required as out that is completly fine. But just not having any way for a mod to have custom shaders is a bit disapointing.

"Executive Summary: Custom shaders added by mods or resource packs are ignored by Iris when an Iris shader pack is loaded. Mods and resource packs using custom shader behavior / vertex formats are extremely difficult to accomodate within Iris."

Is stated as the main reason for irises decision, so im asking why not force the mods into specific formats and for the mod to implement a shader that stays within predefined formats? Also, if a Shaderpack has a Shader for the same thing, just use the shaderpacks.
Im specifcly asking about the "with iris my stuff turns invisible" issue, not for resourcepacks overwriting vanilla shaders.

This is not possible. The reason why is pretty complicated, but it comes down to the fact that the buffer system is user specified.

Iris gives colortex0-15 for shaders to use. These can be used differently in every shader, and the output is combined by the shader in compositing pass. You could render to colortex0, but there's no guarantee colortex0 will be read by the shader ever!