I know this would likely require custom textures for every fish in your mod, but it seems weird not being able to cook them like vanilla fish.

Perhaps in the meantime you can add one "cooked fish" item that all of them smelt into?

We have this, its the filet system.

You use a knife to get a number of filets based on the type of fish. Then you cook those.