When I change the destination it'll work for about 10 seconds then just disappear. Here are the commands I'm using: /portal set_portal_destination minecraft:overworld -272 101 186 and I have no other mods.

see this https://github.com/qouteall/ImmersivePortalsMod/wiki/Portal-Customization#editing-existing-breakable-portals