Add a modifier for the pots speed and chance of drop.
(alternatively you could simply override the Growth rate)

It could be global (applies to every item), category specific, mod specific or item specific.
This could be set through the configs or directly with an in-game command (server friendly) and the recipes should be auto-updated.

For example, you could have the global multiplier set to 1.25, with a specific multiplier for some items which overrides the global value

Example of commands
/bpots speed global 1.25
/bpots speed minecraft:acacia_sapling 1.75

and is possible in version 1.16.5-7.1.39

All the content in this mod is provided through Datapacks, this is already a server friendly way to add/remove/modify the content. Editing lots of JSON files can be tedious but this is why the community has created tools such as CraftTweaker to allow more advanced modifications.

For example, with the latest version of BotanyPots this script can be used to make every crop take 20x longer to grow.

import mods.botanypots.ZenCrop;
val crops = <recipetype:botanypots:crop>;

crops.forEach((recipe) => {
    
    recipe.setGrowthTicks(recipe.getGrowthTicks() * 20);
});