Since the portal charm doesn't work with https://www.curseforge.com/minecraft/mc-mods/randomportals
Would be cool to have such a config option

The portal charm affects the player's portal timer value. Any portal not compatible with the charm is tracking the portal timer in their own non-standard way. These issues have nothing to do with which blocks the player enters and a whitelist would not help or make sense here as there is no portal list to begin with. The proper fix would be to ask that mod to use the standard player portal timer.