Going downwards with the phantom origin is very slow and nothing can make it faster currently, but with an elytra and fireworks you can go up and horizontally very quickly. I think adding a secondary power keybind to let you freefall phase would improve the quality of life for the phantom, maybe it would use more hunger for balancing?

You'd just end up splattering, wouldn't you? Unless you take it very carefully and only go down 2 or 3 blocks at a time.
The Phantom still has fall damage, so would it only apply if you leave blocks? Could you hold the hotkey to avoid hitting the ground and just phase in to it a bit and hop back up, no fall damage taken?
Or would it only be able to be used if you're touching the ground when you start pressing the hotkey, and then automatically cancel if you hit free air?

Yeah you would immediately die if you don't have something to negate fall damage from another mod or an elytra. I mainly want it for the elytra.

I figured out that this is possible to implement in a datapack

Oh?
Will you be uploading that datapack somewhere?

Nah, the datapack does a bunch of other stuff i just realized I could remove the condition when making it

The datapack is made for my (private) modpack and is built around the phantom evolving for some qol stuff that both doesn't make sense outside of my modpack and would be way too overpowered outside of it. The phantom evolves to swim in lava like water (no fire resist because cold sweat and silent gear mean you would probably have it already) which I only stole from another origin datapack because I'd been wanting it for a while. After the dragon the phantom no longer burns in the day, but has 2x exhaustion (silent gear crimson steel armor gives fire resist and is before then in progression), And after the alex's mobs void worm it gets the noclip phasing.