I may not understand something, but how I should combine experience modifier and caster`s gauntlet? (casters gauntlet have elemental modifier terra with effect cast speed)

You have to craft the experience modifier with an effect crystal first, either speed, vis cost or power (this will select what the augment will improve). Once you've done that you can then attach it to the gauntlet.

As an additional question, you say you're using an elemental modifier, so did you put an effect crystal on that too or were you able to attach it directly to the gauntlet?

I think just adding a note in thaumonomicon at experience and aggression module that you still need a speed cost or power crystal would be a great help