coz this is how skeletons work in Minecraft.

The same thing would have to apply for Strays and Pillagers, as well as Piglins that are wielding a Crossbow.

The amount of arrows dropped (or any item for any mob) could be specified using a loot table specifying additional items to add to the mob's inventory upon first possession. An example loot table could drop:

• Skeleton upon first possession: 1-5 normal arrows (that actually lasts longer than what the numbers say due to bonuses)
• Stray upon first possession: 1-5 arrows of Slowness I
• Pillager upon first possession: 2-7 normal arrows
• Piglin wielding Crossbow upon first possession: 10-22 of a mix of normal and spectral arrows

Values were taken from existing loot tables (mob loot, pillager outpost, and bastion generic chest respectively).

On one hand yes, on the other that would be stupidly powerful early game. What we can probably do is add a few arrows to the skeleton's inventory.