Would it be easily possible to add a simple API for mod devs that would like to include their "books" that don't fit any of the patterns already included? Or would it just be easier to include it via config?

This would be easy enough, just adding an empty interface IModdedBook and adding the following to
AttachmentRecipe#isTarget

if (stack.getItem() instanceof IModdedBook){
    return true;
}

Capabilities would probably be a better idea, less hard dependencies.