Lucid Technologies & Solutions Pvt. Ltd.
Scaling governance to address the growing inventory of data assets enabling effective use of data and improved productivity
Business Problem
• Client was looking to curate and manage a rapidly growing inventory of data assets comprising over 90 thousand tables, views and reports with a team of 40 business and technical stewards, to enable effective use of data and improve productivity
• Another key objective of the data governance program was to drive efficiency by enabling users to find and reuse data and analytics content across the enterprise before developing new content
• These challenges had to be addressed with limited resources
Lucid Approach
• First was to enable automated ingestion of technical metadata and enable lineage at the attribute level across cloud and on-prem BI platforms, Analytics platforms – Cube stores, in-memory DB, Big Data platform and the enterprise data warehouse
• In order to support the certification and watermarking of data and reporting assets, the Client needed extended metadata to be captured for assets and used as criteria for certification. For example: Annotations set in source platforms was extracted and used to link data assets (technical metadata) to glossary assets in Collibra
• Lucid’s Easy Connect metadata integrations for BI platforms, Analytics Platforms and Big Data platforms with custom extensions were used to support extended metadata, enterprise security, scalability and high availability requirements
• Lucid then built a bespoke solution comprising of custom workflows for certification and watermarking of data assets and report assets • The Certification process comprised of a technical review to ensure use of appropriate data (based on lineage), adherence to development standards(availability of extended metadata) and effective and efficient use of resources by using an automated process to identify “similar” data assets and reports and assessing them using “match scores” calculated based on glossary links • Lucid and the Client also collaborated to develop a new solution for a Search API to scan Collibra assets and return results based on relevancy rules • A Compliance Dashboard prototype was also developed to measure the compliance levels of the assets and the success of the Governance processes implemented • Workflows were built to identify redundant data assets and archive deprecated assets
• Lucid’s Easy Connect metadata integrations for BI platforms, Analytics Platforms and Big Data platforms with custom extensions were used to support extended metadata, enterprise security, scalability and high availability requirements
• Lucid then built a bespoke solution comprising of custom workflows for certification and watermarking of data assets and report assets • The Certification process comprised of a technical review to ensure use of appropriate data (based on lineage), adherence to development standards(availability of extended metadata) and effective and efficient use of resources by using an automated process to identify “similar” data assets and reports and assessing them using “match scores” calculated based on glossary links • Lucid and the Client also collaborated to develop a new solution for a Search API to scan Collibra assets and return results based on relevancy rules • A Compliance Dashboard prototype was also developed to measure the compliance levels of the assets and the success of the Governance processes implemented • Workflows were built to identify redundant data assets and archive deprecated assets
Results
• Governance scaled to address the growing inventory of data assets enabling effective use of data and improved productivity – a team of 40 business and technical stewards curate and manage a rapidly growing inventory of currently close to a hundred thousand tables, views and reports
• Operational efficiency in finding and reusing data and analytics content across the enterprise before developing new content. Also identifying redundant assets and archiving the deprecated assets
• Continuous monitoring of the effectiveness of the Governance processes to ensure that the data remains ‘meaningful’ to the consumers
Lucid’s industry leading perspectives on managing and curating the data in DGC have formed the cornerstone of our Enterprise data management processes. Working with them has accelerated our ability to reuse our data capabilities.
Director of Value Stream Data Architecture and Lead Architect for the Enterprise Catalog
Semiconductor industry