Instead of tracking it on demand on each pulse, rather make an event listener to build the buffs table (on aura applied / removed).
Player is the top priority but the handler should be flexible to take any UnitIDs/GUID (so, in the future it could be extended to UnitIDs even if the GUID changes, which is something that can happen with nameplates tracking especially).

@EvanMichaels did a first step towards this direction in 78228fc . Might re-use that for Auras since it's one of the other big resource consumer.

Still planned for Shadowlands development.
Talked with WA folks to know if they were interested to make their internal tracker an external lib in order to improve overall addons performance since it means that each addon has to reimplement the same system (and thus duplicate both the processing power and memory storage requirements).
They are not interested so I will either make it its own lib and use it inside HL or just add it into HL.