I've noticed that when I play on servers that has me switching worlds every time it does it, it loads at least half a gb of memory that doesn't go away unless I restart mc entirely.
The issue seems to only be with bobby installed.

I had the same issue but on switching dimensions on my server, uninstalling bobby fixed it. without bobby memory resets to a baseline of ~1Gb no matter how many dimension switches occur, with bobby it too starts at 1Gb but increases by the already mentioned 500mb - 1Gb every dimension switch never going down to the baseline even after a manually trigger cleaning cycle, quitting the world or just waiting. I found that changing resource packs helps for some reason, though i cannot confirm that that isn't just incidental.

Cannot reproduce.
Here's what I did:

1. Set Singleplayer Server View Distance in Bobby options to 12
2. Set Render Distance to 96
3. Create new creative world
4. Fly around until around 1400 chunks are loaded by Bobby
5. /execute in minecraft:the_nether run tp @p ~ ~ ~
6. /execute in minecraft:overworld run tp @p ~ ~ ~
7. Take note of minimum memory usage value in F3 (~950MB for me)
8. Repeatedly run 5 and 6 for ~20 times
9. Look at min memory usage again, still ~950MB for me