After my testing, I think the probability of killing creatures and dropping bags is very high.

I want the hat to become a rarer item, so can I modify the probability of a creature dropping the hat?

Where should I change the configuration?

Just as with every CF comment and github issues like #18 , either use the simplified config in the Fabric version accesible through ModMenu or the config files, or use a Datapack for Forge/Fabric to modify drop rates like a normal loot table.

With all due respect your attitude on the subject is rather misguided - I implore you to consider adding instructions on how to adjust the drop rates of hats to the front page of your mod rather than pointing at peoples' issues on the subject as "every comment like this" cases.