Now this might sound insane, but could OptiGUI work with Linked Storage and make replacements based on the color combination & if the chest is private or public?
The 3 color slots with 16 colors already makes up 816 combinations and that x2 if the chest is private or public makes up 1632 combinations. Would be insane doing those textures, but apart they also being optimized and resource pack also being released as separate addon of the main resource pack I might go for it if OptiGUI could support it.

UPDATE: had some thoughts on it and this would be more for a png slice/overlay which is introduced with the upcoming minecraft version (1.20.2+) as making 1k complete GUI textures wouldn't be a wise idea.
therefore if a png sprite could change its texture which then gets put with .mcmeta on the actual GUI which texture does not change - short: just the sprite changes depending on the color combination
hope I explained it properly ... also if something like that would be even possible

I don't know how overlays work, I'll look into it when 1.20.2 Pre-releases/RC versions drop. The core part of OptiGUI (TextureReplacer.kt) may need to be changed to support this.
Sounds easier than making QuickShulker compatible, which is an ongoing issue (probably since OptiGUI 2 was made). I finally made it compatible (code not yet pushed), I'll write it in #19 why it wasn't straightforward.

Just a side note regarding the QuickShulker issue: I did build QuickShulker with that commit of yours and it does work with the current released OptiGUI version. Haven't tested all the Shulkers, but if one works all of them very likely work. Also haven't tested all use cases and also didn't checked further for any issues

(Sorry about the off topic, just had to clarify this)

The first alpha builds of OptiGUI 3 are finally here for 1.20 and 1.21!

OptiGUI 3 uses NBT instead of selectors, which is way more compatible with other mods out-of-the-box. Press F12 while the inventory is open, and OptiGUI will generate and copy a JSON Resource to your clipboard.

A correction about your math, 16 colors for 3 channels is 16*16*16=4096 textures (x2 if private/public).

The 1.20.2 overlays are not multiple textures on top of each other, but multiple directories, and the game chooses based on the game version.

Rendering multiple textures instead of one is easier said than done, especially before 1.20, tho I may try it in the future.