Current (Carpet Mod 1.4.40 for Minecraft 1.17), some entities that spawn during world generation may not get their entity_load_handler applied, meaning that whether code is inside entity_load_handler for that entity type does not run for these worldgen spawned entities. To reproduce:

1. Load a Scarpet app with the following code.
2. Explore some new chunks.
3. Notice that the number in the status bar (where it says "There are # untracked entities") will increase from 0.
4. Run /say @e[tag=!TEST-1].

Full source code of test app:

__config() -> {
	'stay_loaded' -> true,
	'scope' -> 'player'
};

__on_start() -> (
	entity_load_handler('*', _(e) -> run(str('tag %s add TEST-1', e~'command_name')));
	for(entity_selector('@e[tag=!TEST-1]'), run(str('tag %s add TEST-1', _~'command_name')));
);

__on_tick() -> (
	player = player();
	untracked_entities = length(entity_selector('@e[tag=!TEST-1]'));
	display_title(player, 'actionbar', str(
		'There are %s untracked entities',
		untracked_entities
	));
);

A workaround is to use a scoreboard tag to denote which entities have been initialized by the app, and initialize any uninitialized entities on a every-tick basis instead of with entity_load_handler, but that workaround can be computationally expensive for large numbers of entities.

to confirm - it only happens in 1.17, right?

I did the same test in 1.16.5 after updating Carpet mod to 1.4.40 in both versions, and the bug does not occur there.

I can see why. Thanks for confirming

ok, that's weirder than I thought.
its a vanilla thing (bug - probably not).

So I moved the even to a more appropriate location for 1.17, but still was getting leaks, and these leaks indicated that they come from mobs from world generation.

Seems like in 1.17 in world generation, entities are only added as their nbt representations, and then 'recreated' when their chunk is fully generated and loaded to the game. This means that if you add a tag to them at that point, since they are already serialized, they will not be affected when being added to the world later. I will dig deeper into that. but for example the following code doesn't report any leaks while yours still does.

global_seen_entities = {};

__on_start() -> (
   entity_load_handler('*', _(e) -> global_seen_entities += e~'id');
   for(entity_list('*'), global_seen_entities += _~'id');
);

__on_tick() -> (
   player = player();
   all_entities = entity_list('*');
   display_title(player, 'actionbar', str(
      'There are %s untracked entities, seen %s, current %s',
      length(filter(all_entities, !has(global_seen_entities, _~'id'))), 
      length(global_seen_entities), 
      length(all_entities)
   ));
);

i will be digging deeper into that because I don't like what I see

There is still a problem - in 1.17 chunk generation with entities is weird - entities come already serialized and there is no way to affect that without changing these optimizations. For now, not gonna sweat about it, but I would suggest keeping that ticket open - maybe there is a better way of handling it. It seems they might be added twice, in this case we might need to disable handling of worldgen entities, if they are then added second time anyways.
Considering the following app:

global_seen_entities = {};

__on_start() -> (
   entity_load_handler('*', _(e, new) -> (run(str('tag %s add TEST-1', e~'command_name')); global_seen_entities += e~'id'));
   for(entity_selector('@e[tag=!TEST-1]'), run(str('tag %s add TEST-1', _~'command_name')); global_seen_entities += _~'id');
);

__on_tick() -> (
   player = player();
   no_tag_entities= entity_selector('@e[tag=!TEST-1]');
   untracked_entities = length(no_tag_entities);
   unidentified = length(filter(no_tag_entities, !has(global_seen_entities, _~'id')));
   all_entities = entity_list('*');
   
   display_title(player, 'actionbar', str(
      'There are %s untagged, %s unseen entities',
      untracked_entities, unidentified
   ));
);