Setup modern data platform

moderndataplatform

Client

One of the largest stock exchanges

Our client is one of the largest stock exchanges in the world by market capitalization. With more than 5,000 companies listed.

Challenges

Project Scope

The size and scale of the company's operations posed major challenges viz.

  • Exponential data growth
  • An inability to carry out complex analytics quickly
  • The information layer was scattered across multiple systems, requiring consolidation.

Datametica implemented the project in three phases.

Phase 1

Enterprise Data Hub

Commissioning a Big Data Platform and consolidating terabytes of data into an Enterprise Data Hub (Go Live Nov 2013)

Approach

Hadoop cluster

The main aspect of the solution consisted of a 150 terabyte Hadoop Cluster to move the traditional information management platform to an open source one.

Phase 2

Migration to Hadoop

Migrating all the reporting and adhoc analysis to Hadoop using an open source reporting platform

Solution

Data migration & EDT

Datametica proposed and delivered the following:

  • Creation of an Enterprise Data Hub and consolidation of data silos
  • Migration of Reporting and Analytics to this framework using open source technologies
  • Migration of the statistical modeling framework from proprietary technology to Open Source R

Phase 3

Migration to "R"

Migrating Predictive Modeling for Fraud Detection to ‘R’

Solution

CMS

Datametica proposed and delivered the following:

  • Creation of a comprehensive CMS (Content Management System) to replace the proprietary legacy system
  • Use of this cluster as an intelligent archive, eliminating expensive storage in the form of emails, documents, backups, and archives.

Benefits

Cost savings

Savings amounted to $150K in annual software maintenance and $1M in future investment to maintain the projected growth rate for the next three years.

Customer experience

The seamless transition to the new data platform was implemented so efficiently that end-users did not experience a single glitch.

Performance

The solution gave our client a much needed, highly scalable environment, resulting in 4x improvement in query performance. It is now possible to run historical queries spanning many years using very complex algorithms.

Operational efficiency

With the introduction of algorithm trading in the Indian stock markets, the trade to order ratio has gone up 125 times. Our solution helped the client bag the NetApp- BW|Businessworld Innovation Award in the Big Data Innovation category in March 2014.

Top