Does it have to be downloaded on both the client and the server or is the server enough ?

Dungeon crawl does not do anything on the client, but it is required to be installed there nontheless. That is because Structures (pyramids, dungeons, etc..) are registry entries since Minecraft version 1.13, and all registry entries need to be present on both the client and the server.
So no, it does not work server-side only and there is no way to change that at the moment.