Databricks Data Engineer

Overview

The Databricks Data Engineer will be part of the Data Services team and help transform the delivery of data driven insights at scale. In this role, they will design and engineer robust data pipelines using technologies like Databricks, Azure Data Factory, Apache Spark, and Delta Lake. This role will work hands on crafting healthcare data solutions - processing massive healthcare datasets, optimizing performance, and ensuring our data is accurate, secure, and accessible when it matters most.

Essential Functions

• Translate business requirements into technical specifications and document solution designs, data flows and architecture

• Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory, Databricks and Apache Spark

• Implement Delta Lake architecture for reliable data storage and processing

• Build and optimize data workflows using Databricks Workflows and Jobs

• Develop scalable data models following medallion architecture (bronze, silver, gold layers)

• Implement Unity Catalog for data governance, access control, and metadata management

• Create and maintain Databricks notebooks for data transformation and analysis

• Optimize Spark jobs for performance and cost efficiency

• Implement data quality checks and validation frameworks

• Collaborate with BI developers, data analysts, and data scientists

• Design and implement data orchestration workflows using Azure Data Factory to coordinate complex ETL/ELT processes

• Develop and maintain CI/CD pipelines for data workflows

• Monitor data pipeline performance and troubleshoot issues

• Document data processes, architectures, and best practices

• Ensure compliance with data security and privacy regulations

• Provide support for new and existing solutions

About You

Knowledge/Skills/Abilities/Expectations:

• Excellent problem-solving and analytical skills

• Strong oral and written communication abilities

• Self-motivated with ability to adapt to new technologies quickly

• Team player with ability to work independently

• Detail-oriented with strong organizational skills

• Ability to manage multiple priorities and meet deadlines

• Experience communicating technical concepts to non-technical stakeholders

Technical Skills: