they said "key could not be linked, set key limit exceeded!"

Hi, please provide more details regarding what lead to this error, and make sure you have read instructions on Castle Doors, in case you haven't: https://github.com/jusipat/CastleBlocks/wiki/Castle-Doors

I connected redstone to the castle door in creative mode, but it didn't open. I also tried using the key."

they said "key could not be linked, set key limit exceeded!"

This means you have used too many keys, assuming you have paired the maximum number of keys and lost them, you have to get a new door. Just to be sure please provide what version you are on so I can test this