Basically, I'm worried about whether or not this could be an issue on servers with large numbers of spells loaded. Is it an issue, or am I being paranoid?

a REFERENCE not an instance. references are tiny, so no, it shouldn't be an issue. I might add that they don't even store references, let alone instances of spells. They store one reference of the current spellgroup they are working with, which means they store ONE reference, which is very little data, no matter how many spells there are.