I think you should be able to move affixes from tomes on lower rarities to higher rarities. For example, you could put a tome with a common affix into an anvil with a rare affix tome to make it a rare affix tome with the common affix. That way in case you used an affix tome of a lower rarity and want to apply it to a higher rarity item, you can. Basically it would be the same as using a rarer affix tome on it at the beginning, just losing the more common affix tome in the process. It would be balanced and I think a pretty reasonable thing to include.

The only importance of the rarity of the tome is what item you can extract affixes from. You can apply affixes to any valid item from any tome.