When entering any of the structures, especially the underground ones, it lags extremely hard, it doesn't crash but the error given is:

[21:36:59] [Render thread/ERROR]: 65539: Invalid key -1
[21:36:59] [Render thread/ERROR]: ########## GL ERROR ##########

Will include the log file as well.

latest.log

Have you tried running with only WDA installed? I see some other mods erroring out too, including:

[20:48:54] [Worker-Main-3/ERROR]: Unknown model loader: twilightforest:giant_block
[20:48:54] [Worker-Main-3/ERROR]: Unknown model loader: forge:composite
[20:48:55] [Worker-Main-3/ERROR]: Unknown model loader: twilightforest:castle_door
[20:48:57] [Worker-Main-3/ERROR]: Unknown model loader: forge:item_layers
[20:48:58] [Worker-Main-3/ERROR]: Unknown model loader: chiselsandbits:chiseled_block
[20:48:59] [Worker-Main-3/ERROR]: Unknown model loader: chiselsandbits:bit

Also try updating your NVIDIA driver since a new driver exists way after you posted this issue. With only WDA and Sodium installed, Issue doesn't happen even on my underpowered hardware (Linux 6.2.2, Mesa 23, Intel HD 4400, Radeon HD 6570)

Since this was 3 weeks ago, I have since then already updated my drivers and also have done what you've referenced

https://mclo.gs/4f9QDpq

I don't know what those errors exactly have to do with lagging in a structure which is confusing, but here's a more recent log with errors highlighted

Have you tried running with only WDA installed? I see some other mods erroring out too, including:

[20:48:54] [Worker-Main-3/ERROR]: Unknown model loader: twilightforest:giant_block
[20:48:54] [Worker-Main-3/ERROR]: Unknown model loader: forge:composite
[20:48:55] [Worker-Main-3/ERROR]: Unknown model loader: twilightforest:castle_door
[20:48:57] [Worker-Main-3/ERROR]: Unknown model loader: forge:item_layers
[20:48:58] [Worker-Main-3/ERROR]: Unknown model loader: chiselsandbits:chiseled_block
[20:48:59] [Worker-Main-3/ERROR]: Unknown model loader: chiselsandbits:bit

Also try updating your NVIDIA driver since a new driver exists way after you posted this issue.
With only WDA and Sodium installed, Issue doesn't happen even on my underpowered hardware (Linux 6.2.2, Mesa 23, Intel HD 4400, Radeon HD 6570)