Is this mod only compatible with 1.16.4? Since it only works with fml 35 or above.

Yeah, Minecraft 1.16.4 corresponds with FML 35. Minecraft 1.16.3 would use FML 34.
Are you asking for the minimum version to be reduced or to add a requirement for minecraft 1.16.4 to the mods.toml as well?
I'm gonna assume the latter though because the mod is currently developed on 1.16.4 so even if it does run on 1.16.3 it would be luck and not supported?