Many networks run multiple servers sharing permissions data. This works fine until a change needs to be made and all the servers have to be reloaded.

Situations here:

• multiple servers, one proxy
  • proxy as sync server/message broker
  • proxy can act as host for webui as well?
  • can we make configuration automatic?
• multiple servers, multiple proxies
  • separate server, or other message queueing software? (redis, zeromq, etc)
  • manual configuration required

Potential links:

• Apache QPID

networking is hard. Servers go down, IPs change, VLANs vanish..
One of the better solutions I've seen is redundancy. The more, the better.

An example is using RabbitMQ as a pub/sub, and Redis as a pub/sub backup for Rabbit. Hell, you can even throw an SQL table in there if you feel like a little masochism that day. Push messages through both. Attach an ID to a message and push it through both systems, then just check the message ID on the other side against a "recently-received" to ensure you're not accidentally duplicating messages.

Additionally, Redis can act as a datastore similar to SQL. Cache upwards; fetch a value from SQL, then put it in Redis, then to an in-memory cache on the server. Record bad/missing data so you don't try to fetch over and over again, and solved. Mostly.

Implementation detail is the biggest bitch. Create a List<MessageSystem> and a List<StorageSystem> (or similar) and have your storage/messaging implement a common interface to make things easier. Run through the list when you want to fetch/store data. Learned that one the hard way, and it's still costing me a bunch of time. Coincidentally, this will make it easier to implement new storage/messaging systems later on when you discover more that your users will inevitably want because of fucking course they do.