Objective: Retire the legacy data warehouse and move to GCP – and become future ready!
Datametica’s client, an American Sports Goods Retailer, aimed to modernize their existing data warehouse platform using Google Cloud Platform, Confluent Kafka, IBM DataStage 11.7, and MicroStrategy Platform.
This initiative contributed to-
- Migrate from the existing Netezza platform to the GCP platform.
- Migrate existing batch jobs from DataStage to GCP native technologies.
- Introduction of Confluent Kafka as an enterprise-class real-time data ingestion and transport layer.
- Upgrade MicroStrategy from version 10.4 to the latest version of MicroStrategy 2021.
- Redirect MicroStrategy reports to Google Big Query.
Customer Challenges: Meeting SLA’s and Scalability Issues
Our client was experiencing issues in meeting daily executive reports, SLAs and scalability during the peak period with the existing DataStage and Netezza solutions. They wanted to leverage cloud-native technologies offered by Google Cloud Platform (GCP) to improve customer insights, better manage operational costs, improve performance, and become future-ready for real-time use cases where AI can be leveraged.
Solutions: GCP Future State Architecture
Datametica modernized the environment and architecture using Google cloud technology and best-in-class frameworks. The approach centered on converting DataStage code to BigQuery SQL and keeping the data model unchanged to reduce change management.
- Datametica created a new architecture and foundation setup to support GCP
- Used Eagle to perform a detailed assessment of the client’s existing environment
- Converted and repointed DataStage ingestion and transformation jobs to GCP native with Raven
- Built streaming (CDC), batch, and file ingestion data pipelines from Kafka to BQ
- Data pipelines were set up for historical and incremental/delta data loads
- Setup orchestration and scheduling in Cloud Composer.
- Data auditing and data quality frameworks were integrated with Cloud Composer
- Pelican ensured 100% accurate cell-level validation between the source and target datasets
- Integrated BQ and MicroStrategy for Data Consumption.
Client Benefits: Faster Cloud Migration, Enhanced Analytics Capabilities, and Lower Operation Cost
- Huge improvement in data pipeline performance.
- Improvement in operational performance and cost.
- Significant reduction in cost
- Datametica automation drastically reduced the business risk.
- 100% accurate, historical, and incremental cell value validation.
DM Products Used
GCP Products Used
Other Technologies
