So Thaumcraft (TC) has its own ParticleEngine that queues and renders its particles, in contrast to most other mods that just use the vanilla ParticleManager. TC's system is relatively similar to vanilla, but the main odd difference is that TC stores lists of particles per dimension id (the ArrayList<Particle> per fx layer corresponding to the dim id key in particles HashMap). These particles are only ever removed from the queue when they can be processed, which is only when the player is in the dimension matching the id.

Now I'm not sure if this can be called a bug or a design decision, but that means that those particles will stay in memory, retaining references to an old World object that the client will never access again (going back to that dimension means the client just makes a new WorldClient object). This means there is a memory leak of the old dimension's WorldClient until the player visits that dimension again, in which case the list is processed and cleared out. But in the case that the player doesn't visit that dimension again for the session, that means you're retaining memory for an unused World object, which can be tens or even hundreds of MBs.

Unless there is some reason that I don't know about TC's reason for keeping these particles in memory per dimension, I suggest that the lists are cleared for the old dimension on dimension change. This would align with vanilla's particle system, as vanilla only ever keeps and processes particles of the client's current dimension, and clears its unprocessed queues on dimension change (see where ParticleManager#clearEffects is called). I don't have a fully decompiled source of Thaumcraft on hand, so I'm not sure what uses the particlesDelayed queue, but that would also have to be cleared on dimension change. I'd like to know your thoughts on this.
Here is an example HTML file of GC roots from a modpack showing a lot of TC particles keeping an old WorldClient from being GC'ed.
Paths-from-GC-Roots-to-WorldClient.zip (Ignore the non-TC roots, they are also causing the memory leak)

Tested with wisps and teleporting between the overworld and nether, and it looks like the fix works! I don't see any worlds being retained after returning to the title screen.

Yeah, that absolutely seems to be a problem. From taking a quick look, it seems like ParticleEngine is a copy of ParticleManager with the major differences being 6 layers instead of 4, and it always uses the Thaumcraft particle texture instead of using either the vanilla particle texture or the stitched block texture. There is also the particlesDelayed list as you saw, but that just seems to be a simple list that gets contains particles that should be added after some ticks.

This should hopefully be fixed now - I tested ambient grass going from overworld -> nether and it seemed to get all of the particles.