Senior Data Engineer - Azure Databricks

Role : Senior Data Engineer (Databricks)Location : Remote | 4 Days/Month WFO at Noida / Hyderabad / Chennai / Pune / Bengaluru

Experience : 7+ Years

Shift : Normal IST

We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management.

Key Responsibilities :

• Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow

• Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps

• Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling)

• Ensure pipeline dependency management, execution reliability, and operational excellence

• Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture

• Implement data observability, monitoring, and alerting mechanisms across workflows

• Optimize pipelines and workflows for performance, scalability, and cost efficiency

• Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers

• Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent

• Ensure data quality, governance, and security compliance

• Collaborate with data science teams for ML/GenAI data readiness

• Mentor junior engineers and drive best practices for orchestration and pipeline design

Required Qualifications :

• 7+ years of experience in data engineering

• Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow

• Strong programming skills in Python and advanced SQL

• Experience in building orchestrated data platforms with multi-tool integration

• Strong understanding of ETL/ELT patterns and orchestration frameworks

• Experience in cloud-native data architectures (Azure preferred)

• Hands-on experience with monitoring, logging, and pipeline reliability

Preferred Qualifications :

• Experience with event-driven architecture and API integrations

• Exposure to AI/ML data pipelines and MLflow