Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud
Solution Architecture
After thinking about the limited problem statement, it seems that security, scalability, data transformation, messaging, declarative rules, HA and persistence are key. A good architecture style for the solution design would be SEDA See this for reference: http://www.eecs.harvard.edu/~mdw/proj/seda/
The solution should be broken down into a set of stages where a component or a set of components interacts with the data and performs logic. Each stage is connected by queues. An orchestration layer can govern the path depending on the routing logic.
Layer 1: Rules Engine. Fronted with incoming and outgoing queues.
Layer 2: Routing Engine: Choreographs the processes.
Layer 3: Executing Engine: Contains logic and data access.
Layer 4: Presentation: Web Server/App server with logic.
0.1 seconds for data retrieval is enough latency to avoid the complexity of managing a distributed cache. Content caching and Akamai edge cache can be useful instead.
Technologies: Use IBM’s MQ Series to define the queues, clusters etc based on logical names. Install Mule ESB to implement SEDA and routes. Bring in iLog or Aion for the declarative rules engine. Host on internal cloud or EC2.
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud
Solution Architecture
After thinking about the limited problem statement, it seems that security, scalability, data transformation, messaging, declarative rules, HA and persistence are key. A good architecture style for the solution design would be SEDA See this for reference: http://www.eecs.harvard.edu/~mdw/proj/seda/
The solution should be broken down into a set of stages where a component or a set of components interacts with the data and performs logic. Each stage is connected by queues. An orchestration layer can govern the path depending on the routing logic.
Layer 1: Rules Engine. Fronted with incoming and outgoing queues.
Layer 2: Routing Engine: Choreographs the processes.
Layer 3: Executing Engine: Contains logic and data access.
Layer 4: Presentation: Web Server/App server with logic.
0.1 seconds for data retrieval is enough latency to avoid the complexity of managing a distributed cache. Content caching and Akamai edge cache can be useful instead.
Technologies: Use IBM’s MQ Series to define the queues, clusters etc based on logical names. Install Mule ESB to implement SEDA and routes. Bring in iLog or Aion for the declarative rules engine. Host on internal cloud or EC2.
