Data Engineer - Azure Databricks

About the Role :

We are looking for an experienced Data Engineer to join our global engineering team and build scalable, reliable, and high-performance data solutions for the healthcare technology domain.

The role will focus on designing and developing robust data pipelines using Apache Spark, PySpark, and Databricks, while leveraging the Azure ecosystem to process and manage large-scale datasets.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Databricks.

• Build efficient batch and near-real-time data processing solutions for large and complex datasets.

• Develop data transformation workflows using Databricks and Delta Lake.

• Work extensively with Delta Live Tables (DLT) to build reliable and maintainable data pipelines.

• Implement and manage data governance and access controls using Unity Catalog.

• Develop reusable and optimized PySpark/Spark code for large-scale data processing.

• Design data ingestion and transformation frameworks across various source systems.

• Implement data quality, validation, error handling, and reconciliation mechanisms within data pipelines.

• Optimize Spark jobs, SQL queries, cluster configurations, partitioning, joins, and data storage strategies to improve performance.

• Work with Azure-based data services to build scalable and secure cloud data solutions.

• Collaborate with data architects, software engineers, analysts, and other stakeholders to understand data requirements and deliver technical solutions.

• Troubleshoot production data pipeline issues and resolve performance, reliability, and data-quality challenges.

• Implement monitoring and operational practices to ensure data pipelines meet availability and performance requirements.

• Follow engineering best practices around version control, testing, CI/CD, documentation, and deployment.

• Contribute to the continuous improvement of data architecture, engineering standards, and development frameworks.

• Ensure solutions follow appropriate security, governance, and compliance requirements relevant to healthcare technology environments.

Required Technical Skills :

• 9+ years of hands-on experience in Data Engineering or related technology roles.

• Strong hands-on experience with Apache Spark and Databricks.

• Strong programming experience in PySpark and Python.

• Strong understanding of Spark architecture, transformations, actions, partitioning, joins, caching, and optimization.

• Experience working with Databricks Delta Lake.

• Hands-on experience with Delta Live Tables (DLT).

• Experience with Databricks Unity Catalog and data governance concepts.

• Strong understanding of Data Structures and Algorithms (DSA) fundamentals.

• Strong SQL skills and experience working with relational and analytical data.

• Experience working with Microsoft Azure and Azure-based data engineering services.

• Strong understanding of data warehousing, data modelling, ETL/ELT, and data processing concepts.

• Experience optimizing large-scale Spark workloads and improving pipeline performance.

Good to Have :

• Experience with Azure Data Factory (ADF).

• Experience with Azure Event Hubs or similar event-streaming technologies.

• Experience with Apache Airflow or other workflow orchestration tools.

• Experience with advanced SQL and query optimization.

• Experience building batch and streaming data pipelines.

• Exposure to CI/CD practices for Databricks and cloud-based data engineering solutions.

• Experience working with healthcare datasets or healthcare technology platforms.

Qualifications :

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

• Strong analytical and problem-solving skills.

• Ability to work independently on complex data engineering problems.

• Strong communication and collaboration skills.

• Ability to work effectively with global and cross-functional engineering teams.

• Strong focus on code quality, scalability, reliability, and performance.