When I run minecraft without mods and fly around in creative mode about 10 minutes, it usually eats from 1 to 1.5 gig ram, not more. But with quark it start from 1.3 and grows up every time, until eat all memory (I have only 4)! After 1 or 2 hours I usually need to restart minecraft or my computer starts to slow down. I have such issues only with quark and also with shaders, any other 40 mods from my modpack require limited part of memory or not require memory at all.

I don't know how to profile memory. Help requested!

You can always go check every datastructure you have in your mod , 1 by 1. Make sure you delete every single 1 when it is not needed anymore. That also means you should make sure you do not create new datastructeres for the same thing an existing 1 is still there.
I know my answer is not really that much helpfull, but that's all i can say.
Datastructures causes memory leakes when not handled properly.
I am also not aware of any memory leaks from quark right now. i have 8 GB reserved for modded MC.

I don't know if it is related, but using the RedShark mod by Darkhax seems to show that many, many packets are sent from quark. As in the file output for packets is over 90 % quark, making the log file over 40 MB