Related to this bug I reported to ATM team
AllTheMods/ATM-10#1892

I see now that the limits were added on purpose, but the refresh/cooldown appears to be broken, when I get "Omniscient" on my sword, duration of 8s, cooldown of 4s, it won't refresh after 4s have passed, and when 8s have passed it gets to 0s remaining and stays there for quite a while.
At some point, hitting a mob may refresh it, but its not very reliable.