net.minecraft.client.recipe.book.RecipeBookGroup is an enum, so you currently can't add new ones. This would also require hooking into ClientRecipeBook.getGroupsForContainer (or the related logic) to have the groups coupled with custom containers.