For now, there's no other way to leave the worldmap display than using a mouse action or the esc key.
This could be very handy to have the Show worldmap shortcut acting as a toggle.

Actualy, worldmap already have this feature. And it's works fine for me. May be some mod prevents this action.