I’ve seen some video in older versions(with a U-turn bottling machime) that you can fill glass bottles with modded liquids.

But when I tried it myself, the bottling machine’s model had changed, and it seems like it's only capable with filling buckets and certain items(jerrycans and chemical throwers).
Take the wine from Growthcraft for example, it worked perfectly filling buckets with it, but when I feed in glass bottles, it just run through the animation and give out an empty bottle. I tried different liquids and the results are all the same.
It did fill the bottle with vanilla potions though. But not water, this makes the problem worse.

I heard that it might be something to do with the new forge system which makes recognizing containers hard.
Is there anyway to fix it or is it being worked on right now?
If not, are there any ways to fix this myself?

Yes, the FluidContainerRegistry, which once made this bottling stuff very easy, has been removed.

The two options available now are:
Have bottles become proper fluid containers like buckets (this would require someone writing a "Universal Bottle" mod)
Or have specific recipes for things like Growthcraft's wine.

The latter you can achieve yourself by use of Crafttweaker, which allows you to add custom recipes for basically all IE machines.
You can find some documentation on their website, and specific documentation for the bottling machine here:
https://crafttweaker.readthedocs.io/en/latest/#Mods/Immersive_Engineering/CraftTweaker_Support/Bottling_Machine/

Thank you!
Just found out that someone has already reported this but its nice to have much more detailed information about this.