Sometimes I get expertise requests, and here is a challenge.
Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud
I will post my take based on these needs. Also, it will be fun to look back at it after a while as new technologies & solution options evolve.
