one of my players found out they could use a machine to replace blocks inside the compact machine dimension which let them then break free into the void. I was wondering if there was any way to prevent this? maybe making it kill them if they go beyond the dimensions of their cube somehow?

Might want to look at Random Things Specter World. You leave the allowed region an it teleports you back into it.

See https://github.com/lumien231/Random-Things/blob/master/src/main/java/lumien/randomthings/Handler/Spectre/SpectreHandler.java

I've also noticed that the Wither mob is able to destroy bounding blocks.

@XileForce what machine did they use?

EnderTech: Redstone Exchanger and Witchery: brew of Erosion

EnderTech has fixed the issue and i've reported it to @Emoniph for Witchery.
Withers being able to destroy blocks is fixed and will be in the next release.

Awesome