Senior Azure Data Engineer - ETL/Data Warehousing

Job Description :

• Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks

• Develop and optimize ETL/ELT processes for large-scale distributed data processing

• Build and maintain data lake and lakehouse architectures on Azure cloud platform

• Write efficient SQL and PySpark code for data transformation and analysis

• Implement Lakeflow Declarative Pipelines for streamlined data workflows

• Perform performance tuning and optimization of data processing systems

• Manage version control using Git and implement CI/CD pipelines for data solutions

• Collaborate with data scientists, analysts, and stakeholders to understand data requirements

• Ensure data quality, security, and governance standards are met across all pipelines

• Mentor junior team members and contribute to technical best practices

Required Skills & Experience : Mandatory Requirements :

• Azure Data Factory (ADF) : Strong hands-on experience in designing and managing data pipelines

• Azure Databricks : Proficiency in building and optimizing distributed data processing solutions

• Apache Spark : In-depth knowledge of Spark architecture and distributed computing concepts

• Lakeflow Declarative Pipelines : Practical experience with declarative pipeline development

• SQL : Expert-level proficiency in writing complex queries and data manipulation

• PySpark : Strong programming skills in PySpark for data transformation

• Azure Cloud Platform : Hands-on experience with Azure data services and cloud infrastructure

• ETL/ELT Concepts : Strong understanding of data integration patterns and methodologies

• Data Warehousing Principles : Knowledge of dimensional modeling, schema design, and best practices

• Distributed Data Processing : Experience working with large-scale data systems and handling big data challenges

• Performance Tuning & Optimization : Ability to identify bottlenecks and optimize data pipeline performance

• Data Lake & Lakehouse Architectures : Understanding of modern data architecture patterns and implementation

• Git & CI/CD : Experience with version control systems and continuous integration/deployment practices

• Strong experience with Azure Data Factory (ADF) and Azure Databricks

• Hands-on expertise in Apache Spark and PySpark

• Experience with Lakeflow Declarative Pipelines

• Strong proficiency in SQL and data modeling

• Solid understanding of ETL/ELT processes and data warehousing concepts

• Experience building and managing Data Lake/Lakehouse architectures

• Hands-on experience with Azure cloud data services

• Proven experience processing and optimizing large-scale data workloads

• Strong knowledge of performance tuning for Spark/Databricks environments

• Experience with Git, version control, and CI/CD implementation

• Notice Period Immediate to 30days only