you can reproduce this thing
MinecraftForge/MinecraftForge#3710
https://www.youtube.com/watch?v=9BDxq7_ThSk
by following this steps, i spent WEEKS to find this. finally, i found. Baubles causes.
1-install forge 2252 to minecraft 1.11.2, and Baubles 1.4.3.
2-open game, create a world (i enabled bonus chest, and allowed cheats)
3-look your around (i did it, maybe no matters)
4-then press ESC, and minimize the minecraft window. then open the task manager, you will see minecraft will use more RAM second by second and it will increase FOREVER.

and you can reproduce it WITHOUT installing Baubles, just by copying the same world that you created when Baubles installed (this is why i opened issue on forge github)
hope you help. thanks.