I'm using version v2.6.3 and I have this on my BaseTrinket, from which all my trinkets extend:

@Override
public TypedActionResult<ItemStack> use(World world, PlayerEntity player, Hand hand) {
  ItemStack stack = player.getStackInHand(hand);
  boolean success = new PlayerTrinketComponent(player).equip(stack);
  ActionResult result = success ? ActionResult.SUCCESS : ActionResult.FAIL;
  return new TypedActionResult<>(result, stack);
}

Whenever I right click with a new trinket, it equips it, but removes ones that were already equipped.
If I go to the slot where the old item was and try to remove it, its effect gets applied, but the item goes missing. Then, if I try to remove the missing item from the slot, the other trinket gets removed too.

I can try to up a video if words are confusing.

This behaviour does not happen when you equip the item manually.

Turns out removing the use method fixed it. Does the latest trinket implement this already?

Yes, TrinketItem already implements this behavior. It is defined in Trinket.equipTrinket and there are some steps you missed to correctly implementing it which lead to excess items