Fertilized dirt will revert to normal dirt when growing saplings. It appears random, I have seen it turn after one tree and I have seen it take 10 trees to change.

Using RandomThings-MC1.9-3.7

It's a forge bug MinecraftForge/MinecraftForge#2866

So did you check whether this also happens in 1.9.4?

Would appear that it does. Using RandomThings-MC1.9.4-3.7.3