I downloaded the latest version RTG-1.9-0.0.5.jar put it in the plugin folder. Deleted all the worlds and set the level-type=RTG in server.properties. Restarted the server. But nothing rtg related happened. just a vanilla world.
What did I do wrong? Or what should I have done instead? A server install tutorial would be nice in this case!

RTG is a Forge mod. Are you running a Forge server? If so, it's simply a case of dropping RTG-1.9-0.0.5.jar into the mods folder and using level-type=RTG in server.properties

If it's still not generating an RTG world, please post your fml-client-latest.log or latest.log file and we'll try to figure out what's going on.

If you are putting the mod in a 'plugin' directory, then I can only assume you are trying to load the mod as a bukkit/spigot plugin. As Pink said, this will not work.