Using my shurikens in combat and they levelled up, preventing me from picking up all the shurikens still stuck in the walls nearby.

Yes, because technically it's not the same weapon anymore. The same happens if you'd add a modifier while they're on the ground. Better this way than being able to pick up too different tools.