I've seen this mod marked as a server side only performance improvement mod. But some people also told me this benefits client as well. So whats the truth here? Maybe it would be worth to explicitly state it in the description, so people don't get confused. Thanks.

So I went to test it. On my system, initial world generation is sped up by roughly 20%. After loading into the world, generating 32 chunk radius was sped up by roughly 40%. However, world map revealed the generated radius isn't a square 32x32, but a circle instead. And the rest was generated after some time. So this benchmark of mine isn't that much valid. I'm guessing the overall performance increase is closer to the 20%.

I do prefer the circular generation, especially with the changes made in 1.18. Also, it looks like Mojang is planning to go this route as well. I'm happy this mod works with the other performance mods I use (I use all the mods).
The Better Fps - Render Distance mod is supposed to do this as well. But it doesn't work with my other performance enhancing mods. This is most likely the reason as to why that mod doesn't work: "Should work with pretty much all mods, though I did not try all which may due to heavily changing rendering skip the modification done by this mod."
I assume C2ME avoids this issue.

I wasn't able to find any benefits on client side only. But maybe in some extreme scenarios it could help. Just as you pointed out.

Btw. For non-fabric and non-forge use cases, Paper (and forks) should be already doing something similar right?

The optimizations of C2ME is primarily designed for server-side performance.
On client-side it's basically the same story as lithium. It works on integrated servers. And the optimizations written for server-side may or may not have some improvements on the client-side. The size of the improvement may vary.

For example, CaffeineMC/lithium-fabric#345 (just want to avoid unnecessary references) is for optimizing entity collisions inside ladders on server-side (bringing MSPT from 175 down to 15 when thousands of entities inside one block with ladders or vines). But this also have some effect on client-side (from 0fps to ~20fps under the same situation above). (Test results from 2No2Name)

However, world map revealed the generated radius isn't a square 32x32, but a circle instead.

This is the visible area, the generated and loaded area is still a square.
And note that C2ME throttles chunk loading outside of simulation distance which is configurable through config updatesPerTick

For non-fabric and non-forge use cases, Paper (and forks) should be already doing something similar right?

Paper has a system called async chunks which is for async chunk loading and async chunk saving on unload. They also rewrite the IO system (which unfortunately seems to be very sensitive to disk performance and latency) and C2ME also have a rewrite of IO system (since 22w03a and need to be enabled manually in config for now).
Some other forks have some small optimizations from C2ME.
But none of the forks that i know have threaded world generation implemented yet.

For some reason my chunk loading works better without this mod. When I use this mod it has only 2 benefits and one big drawback. The 2 benefits are that the chunk loading is faster and you can go up to 128 BUT. When you travel long distances all of those chunks stack up into your RAM, and when it gets full it has big stutters and makes the game way less playable.

When you travel long distances all of those chunks stack up into your RAM, and when it gets full it has big stutters and makes the game way less playable.

Should I disable if playing offline?