When the Leviathan Axe and Mjolnir return mid-air without hitting a block or mob, they don't go back to the player hand, but instead just circle around the player.
Only way I've found to stop this is to get rid of the entity using the /kill command. This is extremely annoying and makes those weapons useless in survival.

Steps to recreate:
Throw the Leviathan Axe / Mjolnir at something far away, making sure it doesn't hit a block or mob
Wait for it to return
It won't return to the player's hand, instead will start circling the player

Mod version: 0.9.8
Minecraft version: 1.19.2

I may be wrong, but I have a gut feeling it might have to do with loaded/simulated chunks. I usually play with simulation distance set to 5 chunks, so I often see projectiles like arrows, ghast fireballs, wither skulls, etc. freeze in place when they hit the simulation border.

I feel this issue might be due to the same. The axe and the hammer make it to the simulation border and just turn back, but they haven't hit a block or mob so their AI gets messed up... Something like that

I'll look into it.

I seems I am unable to recreate the issue. I believe the issue is that the return speed is too high so it zooms past the player, then, if the player is jumping and running around, it has a hard time returning to the player's location. Either the speed is the issue, or you simply had full inventory. I will reduce the return speed and see if that helps.