On collision with the ground, it seems like the broom remembers it's max height and then damages the player according to that height as fall damage. So you can easily fall from a high place after flying really high and then flying low and suddenly colliding once with the ground.