First off, my specs:
MC Server 1.18.1 / Fabric 0.13.3 / Fabric API 0.46.6 / Mod Version 1.2.18

Graves in the Server config are defined as
generateGraves = true retrievalType = "ON_BREAK" dropType = "IN_INVENTORY"

The graves are unbrekable in survival mode. I've tried bare hands, TNT as well as Netherite Pickaxe with Haste 50 applied. None lead to breaking the grave even after a couple seconds. Breaking the grave in creative does put the "grave replaced with air" message in the server console, but doesn't give back items. In this case, graves are still shown and retrievable through the yigd moderate command.

Maybe I'm missing something, but I wouldnt know how to break it.

If this config is set to "on break", it needs to be set to this on the client too, since the break progress is calculated on client
This should fix your problem, but I'll leave it open for now in case it doesn't work or you have any suggestion regarding this issue

Since there's been well over a month and you have not replied I assume you fixed it. Closing the issue. If you ever experience anything other that you experience as a bug it would be very kind of you to report that. Thanks!