Senior Data Engineer - Azure Platform

Job Role : Senior Data Engineer.

Experience : 6+ Years.

Location : South Bangalore, KA.

Key Responsibilities :

• Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and Azure Data Factory.

• Build scalable batch and streaming data processing solutions using Python and PySpark.

• Implement and maintain Delta Lake and Lakehouse architectures.

• Design and manage data models, including Dimensional Modeling and SCD Type 1 & Type 2 implementations.

• Develop data storage solutions using Azure Data Lake Storage (Gen2, Delta Tables, and Parquet formats).

• Integrate external systems through REST APIs and secure authentication mechanisms.

• Build and maintain CI/CD pipelines using Azure DevOps and Git.

• Implement data governance, security, and access control best practices.

• Optimize SQL queries, data processing performance, and workload efficiency.

• Collaborate with cross-functional teams to deliver scalable and reliable data solutions.

Required Skills :

• 6+ years of experience in Data Engineering.

• Strong hands-on experience with Azure Databricks, Python, and PySpark.

• Experience with Azure Data Factory (ADF) and Azure Data Lake Storage.

• Strong knowledge of Azure DevOps, Git, and CI/CD implementation.

• Experience with REST API integrations and authentication mechanisms.

• Strong SQL skills including query optimization, CTEs, and window functions.

• Experience with Delta Lake concepts including ACID transactions, Time Travel, Vacuum, and Z-Ordering.

• Understanding of distributed Data Processing, Data Governance, and security best practices.

• Experience with Databricks Asset Bundles (DABs).

• Exposure to event-driven architectures and streaming data solutions.