removing connector will prevent many issues, so wouldnt it be good to work on it?

I plan to merge #564 to remove the need for Connector, but I do not plan to remove the dependency on Forgified Fabric API. However, I have not seen a significant amount of issues caused by either of these, so I do not know what you mean by "many issues".

Connector seems to be the highest tick usage on spark

Report that to Connector then, with the full profiling report.