It would be nice to be able to add a new category by lua API, currently I can kind of do it in a roundabout way with the following lua:

local function addCategory(name, search)
    local bc = BAGANATOR_CONFIG
    table.insert(bc.category_display_order, name)
    bc.category_modifications[name] = {}
    bc.custom_categories[name] = {
        name = name,
        search = search
    }
end

but it would be nice to have an official API for it.

Why do you need to add categories by API? If this is about sharing categories there's already an import/export system in the UI.

I want to create it from different addons, like in BetterBags plugin system.

No plans to do this as explained in Baganator/Syndicator#27