The further you explore in a session, the more memory is used. And the amount of memory used can be extremely high.

Possible workaround could be to forget chunks you're super far from after n amount of time?

It already forgets chunks used, so while memory used is somewhat higher it would not explain your high memory use. This mod simply fixes the amount/distance at which the client keeps the chunks to 32(or config value), so if you move past that it'll forget those chunks as normally.