I am working on a modpack, and I'm trying to setup two different quest lines that stem from one main one. Is there a way to make it so that going down one line, completely blocks the other one from being an option to go down?

You can do that by having your two optional quests linked to each other (each is a requirement of the other. The linking tool may not let you do this so use the menus) and then setting them to use XOR logic. The parent quest can remain the same per normal logic.