When I try to connect two panels with one that is vertical and the other one being horizontal, I am forced to use one other tiny redstone containing block which will register the connection state.
-> first it seems to work
-> but then I remove tiny redstone torch on below circuit and I keep the redstone signal in the middle block.

This is related to #114 in that panels of different orientations do not directly interact with each other.
However, the new issue here is that the signal persists when the source is removed.

A temporary workaround for this until #114 is implemented is to add "tinyredstone:redstone_panel" to the "restone_wires" list in the server config. So, line 19 in the tinyredstone-server.toml file (found in the serverconfig folder inside the world folder) would be as follows:
redstone_wires = ["redstonepen:track", "cb_multipart:multipart", "cyclic:clock", "tinyredstone:redstone_panel"]

Thanks for reporting this!