Sunday, September 30, 2012

My response to an expertise request

Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud

Solution Architecture
After thinking about the limited problem statement, it seems that security, scalability, data transformation, messaging, declarative rules, HA and persistence are key.  A good architecture style for the solution design would be SEDA See this for reference: http://www.eecs.harvard.edu/~mdw/proj/seda/
The solution should be broken down into a set of stages where a component or a set of components interacts with the data and performs logic. Each stage is connected by queues. An orchestration layer can govern the path depending on the routing logic.
Layer 1: Rules Engine. Fronted with incoming and outgoing queues.
Layer 2: Routing Engine: Choreographs the processes.
Layer 3: Executing Engine: Contains logic and data access.
Layer 4: Presentation: Web Server/App server with logic.
0.1 seconds for data retrieval is enough latency to avoid the complexity of managing a distributed cache. Content caching and Akamai edge cache can be useful instead.
Technologies: Use IBM’s MQ Series to define the queues, clusters etc based on logical names. Install Mule ESB to implement SEDA and routes. Bring in iLog or Aion for the declarative rules engine. Host on internal cloud or EC2.
 

Wednesday, September 26, 2012

How would you solve for these requirements?

Sometimes I get expertise requests, and here is a challenge.

Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud

I will post my take based on these needs. Also, it will be fun to look back at it after a while as new technologies & solution options evolve.

Sunday, September 23, 2012

A Culture of Discipline

Those who try to manage change, motivate employees and work to create alignment are negatively correlated with companies that move from good to great. To get a company to top gear – a culture of discipline is critical.

Think about it – a CULTURE of DISCIPLINE. This means that everyone does what they are supposed to without stepping on toes, do what they do best and are diligent about the task at hand. No one has to remind them and they produce excellence for the team. And then you get a team of teams, a culture that pervades the organization. This is what is required for an organization that transforms itself.

I am re-reading this book “Good to Great” by Jim Collins. It’s a classic. I really like books that dwell on evidence and then synthesize that into evidence-based recommendations. I have taken a few of those recommendations and internalized them into traits, I present a few in “Traffic Signals” here.

So here are a few traits of Great companies -

image

Red: Negative Correlation (with Good to Great)

Green: Positive Correlation

Black: No difference.

The culture is an enabler to excellence. It is the HOW.

The 3 circles need to intersect to understand the WHAT.

The hedgehog concept revolves around , in my mind 3 questions

Do you love what you’re doing?

Do you excel at what you’re doing?

Does it pay ?

If the answer is true – the you’re excelling and if its true for the organization – it is poised to be great in the future.

A good pre-read is 7 Habits…because it can link the individual habits of successful individuals to the culture required for greatness in an organization.