I only change the Target's Cast Bar position and a few of the pinned taskbar items if i switch characters or even recall to my house, it resets all of my settings.

You will have to try deleting your ForgeUI settings or deleting all addons and slowly installing them back. Unfortunatelly this is bug of WildStar and I can't do much about it.