Real Time Stock Market Data Processing - Design

Use case - Processing real time stock market data like quote & trade.  Generally Stock market publish the updates, intra-day live data, via API's, which an application can listen and publish to its clients.

Flow - Steps 
1. Connect to Stock Exchange server and register for Events (trade, quote. depth).
2. On receiving event, validate event, log event, and pass it on to Handler.
3. Handler knows EVENT_TYPE and handles events appropriately.
4. Handler converts events into domain objects, which are usable by the Business layer or other clients.
5. Business layer generates Reports/Metrics and might make decisions on investment. Business layer might not understand the event, so it is required to map the events into domain objects which are more understood by the higher layers.
6. Domain objects might be distributed to other applications or clients. Other clients may desire objects in their required format, like XML, EDI, etc.

Design Considerations - 

  • MBeans should be created & registered on application startup. MBeans are used for monitoring, like - Memory Consumption, Statistics, Reprocessing and event, Re-sending an message (/domain object) to a client. 
  • We might receive thousands of thousands of events in a day, so we can't process events sequentially. We should maintain pool of handlers. 
  • Events should be logged into Database with event stage. Event stage is optional but it should be different from the EVENT_TYPE & EVENT_STATUS which we receive in the Event. 
  • User details for Stock Exchange Server can be maintained in properties file. 
  • Different Event types can have different structure. 
  • Events should be placed in Queue, before passing to Handler. OR if Java Thread Pool is used, which comes with a blocking queue, where tasks are placed before actually getting handled by a thread. 
  • Events should be validated, possibly for checksum, to ignore duplicate events. 
Flowchart - 

 Detailed Design - 


Popular Posts