I farm Throne of Thunder trash for transmog. I make 20 instances per account with 2 characters on different realm and faction. This mean limit is for faction or per realm. I cant test this why i dont have more chars.

not quite sure about this, so can you share more information about this? Is it said "You have entered too many instances recently" after you clear and reset instances 10 times in a realm or faction?

I have Alliance on Outland and Horde on Kazzak. When reset 10 times on Outland i go in Kazzak and can reset 10 times more.
When go 10 times on alliance realm i am block to enter in dungeons with all my characters on that realm but when try on horde in other realm i can enter 10 more times.
I dont have Horde/Aalliance on same realm.

I can reproduce this with all Alliance characters on different realms, but I don't have both factions on same realm either. I guess we need more infomation to confirm this.

As far as I know, the limit is ten unique instances (resets) per server. If SI counts attempts for all realms towards the total, then that's probably wrong, as it should count them on a per-server basis. Or am I misunderstanding the issue?

Yes, I guess limit is per-realm, but still need confirmation.

In addition, I have related issue when I downgrading my mythic keystone. SI didn't count when I enter, start and reset.

Yes, the limit is per realm and unaffected by the player's faction. I just tested this to make sure, though anything else would've been fairly contrived as it is.