Business Problem: The client was struggling with a non-scalable data integration platform that could not take in the ever increasing data volumes
Our Solution: The work involved performance tuning of data load jobs to load Click-Stream analytics data to support the increased data volumes in the available load window and thereby improve the utilization of the available infrastructure.
This required understanding the custom NZ SQL scripts and effectively rewrite the scripts that were causing load bottle-necks as parallel jobs in DataStage 8.7. The requirement was to improve current load time for source data of about 18 Million Rows of data across 9 tables (~50GB) into a target table of keys (to use in Dimension lookups) of ~3GB from ~25 minutes of a highly tuned NZ SQL procedure to under 6 minutes in DataStage. The project was also able to demonstrate a saving of 60% in CPU and 77% in memory utilization by moving the processing from NZ-SQL to DataStage.
Technology: Netezza TwinFin12, NZ-SQL, DataStage 8.7