A friend tried to grow a sacred rubber sappling (MineFactoryReloaded) and it broke the walls letting us to go out in the compact dimension.

Edit: forgot to say that we use the latest build at this date with Forge 1352

As you already said, i don't think there is much I can do about it. Looks like MFR is not properly checking whether leaves can break the blocks, see canBeReplacedByLeaves.
Should be reported to MFR. Already aware of this, @skyboy?

Erm, sorry, I made a mistake.
It was not a sacred rubber, but just a big one (I disabled sacred on my server since the beginning, it couldn't be this one).

The enchanted sacred rubber tree doesn't remotely consider that: it is world-destructive to grow, save bedrock. I'd suggest filling the entire dimension with some indestructible block instead of air (since you have no windows/transparency, bedrock is fine). This won't stop other trees, like the vanilla dark oak tree, from breaking out chunks of bedrock but they will at least be slower. Since it's the same block, this won't increase the size of the dimension on disk either.

Currently working on a 1.10.2 version, the 1.7.10 version is not being maintained anymore, sorry.
Closing all issues not relevant for the new version.

I think this is a vanilla bug. You can also break the bedrock in the nether with a tree in vanilla minecraft.

Yeah, it's fairly possible. Thus it's not possible to do anything, I think