When I press R key with a modded mob that has ability, nothing happens (Mob tested: Sun, from the Dungeons Mod Lite mod)

Unlike animations & visuals which can be done automatically, abilities require me (or the other developer) to add specific integration for entities. If you're looking to add abilities for a modpack and know a developer, the AbilityRegistry class makes it easy to integrate with Identity.

In other words, only select vanilla mobs will have abilities in Identity. A mostly complete list of supported mobs can be found here. Adding an expansion/external mod to support other projects would be possible, but is not something I plan to do.

I tested enderman but it didn't work

If you have enough coding knowledge, then you should be able to make a custom Identity ability for the mob(s).