I was able to create a [Buy] sign selling 3456 items, using a double chest.

This trade is not possible because double chest has more item space than player inventory.

SignShop should give some kind of error (or not register blue sign) on trades that require more inventory slots than player inventory.

I don't really think it should. It's not like the plugin allows the trade to go through or something. It could also be that some server or something has a plugin that allows players to have more than normal inventory space, or allows items to be stacked higher than the normal amount and the transaction would succeed. So really, we don't have a good way of knowing those things, and should just check when a player tries to do the transaction. I think it's up to the person creating the sign to determine how much they want to sell at a time, not us.