I've noticed that when I play on servers that has me switching worlds every time it does it, it loads at least half a gb of memory that doesn't go away unless I restart mc entirely.
The issue seems to only be with bobby installed.

I had the same issue but on switching dimensions on my server, uninstalling bobby fixed it. without bobby memory resets to a baseline of ~1Gb no matter how many dimension switches occur, with bobby it too starts at 1Gb but increases by the already mentioned 500mb - 1Gb every dimension switch never going down to the baseline even after a manually trigger cleaning cycle, quitting the world or just waiting. I found that changing resource packs helps for some reason, though i cannot confirm that that isn't just incidental.

Cannot reproduce.
Here's what I did:

1. Set Singleplayer Server View Distance in Bobby options to 12
2. Set Render Distance to 96
3. Create new creative world
4. Fly around until around 1400 chunks are loaded by Bobby
5. /execute in minecraft:the_nether run tp @p ~ ~ ~
6. /execute in minecraft:overworld run tp @p ~ ~ ~
7. Take note of minimum memory usage value in F3 (~950MB for me)
8. Repeatedly run 5 and 6 for ~20 times
9. Look at min memory usage again, still ~950MB for me

Sorry for the late reply,
after a lot of further testing I pinned it down to ETF (Entity Texture Feature).
I created a new Profile, installed only Bobby, Modmenu and ETF. I copied your render distance settings. Tested it on a server hosted locally on my PC. I traveled between Overworld and Nether in multiple places (I think this is Important). Every portal was out of the 96 render distance in the Overworld. after 5-6 times my assigned 8Gb are filled.
After posting and re-reading i thought a video might be helpful so here is a link: https://vimeo.com/1039283781?share=copy
I hope this helps with finding finding the Issue. Just tell me if something else would be helpfull aswell.

I'm affected as well