The Challenge
Fastest growing bank in the region – huge volume of data, variety of sources. Complexity in terms of managing, securing and using it to generate useful insights that improve business decisions. Requires the right set of technology and tools that can be real-time with the same pace at which the data is generated. The biggest challenge being the platform refresh and move out from legacy in terms of technology, skills, processes etc, that enable happy customers for the bank.
The Approach
- Decision to move from traditional to open source big data technologies.
- Use of dynamic templates to create the data lake and for supporting Mass Data Ingestion.
- Using industry standard model for building the warehouse leveraging cutting-edge technologies.
- More focus on data governance and profiling.
- Real time data processing.
Technologies
- Big data technologies like Hadoop, Spark, Hive, Oozie, Sqoop, etc.
- Informatica BDM, Informatica PowerCenter.
- SAP FSDM (Financial Services Data Model).
- Informatica EDC, CDC and MDM.
- HBASE, Flink, Beam, Kafka.
- Jenkins, Shell Scripting, GitHub.
Enterprise data, Data quality, Governance & management challenges
- Managing enterprise data.
- Change management.
- 360° view of customers, products.
- Data security & ownership.
- Regulatory & international standards.
- AML ( Anti Money Laundering).
- Poor data integration.
Solution
- Identifying Enterprise Masters.
- Defining the data ownership and lifecycle.
- Creating and maintaining an Enterprise Business Glossary.
- Periodical data quality monitoring of masters.
Setting up a data governance council
Tools Used
- Data governance – Informatica Axon.
- Data quality – Informatica Big Data Quality.
- Data lineage – Informatica EDC.
- Master data management – Informatica MDM.
- Customer data screening – Fircosoft Trust.
