Iitjobs Inc.
Senior Databricks Data Engineer
- Compensation
- ₹1.5M–₹3M / yr
- Location
- Mumbai (All Areas)
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Full description
About the Role:
We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines.
- Develop PySpark and Spark SQL transformations.
- Build Bronze, Silver, and Gold layer data pipelines.
- Integrate SAP and enterprise source systems.
- Implement Delta Lake best practices.
- Optimize Spark jobs for performance and cost.
- Build reusable data engineering frameworks.
- Implement data quality and validation rules.
- Develop incremental and CDC pipelines.
- Collaborate with BI developers for optimized reporting datasets.
- Participate in code reviews and technical documentation.
- Support production deployments and troubleshooting.
Roles & Responsibilities
Ideal Candidate
- 812 years of Data Engineering experience.
- Strong hands-on experience with Azure Databricks.
- Expert in PySpark and Spark SQL.
- Experience with Delta Lake.
- Experience with Azure Data Factory.
- Knowledge of ADLS Gen2.
- Strong SQL skills.
- Experience with Git and CI/CD.
- Understanding of SAP data extraction techniques.
Preferred Certification
- Databricks Certified Data Engineer Professional.
- Experience with Structured Streaming.
- Knowledge of MLflow.
- Azure DevOps experience.
- Experience with Power BI datasets.