Ali Raza

Recognized with a Spot Award in May 2023 for optimizing the gold layer data ingestion process, resulting in a remarkable 25% reduction in overall processing time. Successfully led the migration of a legacy system for a prominent global corporation to cutting-edge cloud-based technologies. Redesigned the solution, achieving an impressive 373% enhancement in overall time complexity reduction Designed and implemented data pipelines for over 5 distinct data sources using Spark and Python, streamlining the data processing and analysis workflow. This effort culminated in a substantial 50% reduction in ETL job execution time. Collaborated on major big data projects, harnessing the power of Spark, Azure Databricks, Synapse, and Azure Data Factory for parallel processing of datasets exceeding 100 million rows. Engineered intricate data models and relational databases, establishing efficient data flows for the parallel processing of information sourced from more than 5 distinct sources. Expertly managed data onboarding from various systems, including Teradata, Delta Lake, Azure Blob Storage, and Azure Data Lake Storage (ADLS). Proficiently conducted Extract, Transform, and Load (ETL) operations on data, leveraging Azure resources such as Azure DevOps, Databricks, Data Factory, and Synapse. Developed a generic process for data cleaning, featuring multiple layers, as well as a versatile pipeline with capabilities including data archiving, email notifications, parallel processing for the same data source, and ETL automation on source data. Played a pivotal role in enhancing data security for Azure cloud-based systems by: Designing a secure process to control access to data stored in storage accounts. Creating a robust process for establishing secure connections between different Azure resources, including Databricks, Data Factory, and Storage accounts. Setting up Azure Purview to enhance data flow visibility for clients and facilitate their ability to request necessary access.