Senior Azure Databricks Data Engineer
- Compensation
- Not disclosed
- Location
- New York, United States
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 4+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
- 9-15 years of experience in Data Engineering and Data Warehousing.
- Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
- Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
- Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
- Experience in handling large-scale structured and unstructured datasets.
- Strong analytical, problem-solving, and troubleshooting skills.
Nice to have
- Microsoft Certified: Azure Data Engineer Associate (DP-203).
- Databricks Certified Data Engineer Associate/Professional.
- Experience with Snowflake, Power BI, or Microsoft Fabric.
- Experience in real-time streaming solutions using Kafka/Event Hubs.
- Exposure to Data Governance and Master Data Management initiatives.
- Microsoft Fabric
- Power BI
- Azure Event Hubs
- Kafka
- Machine Learning data pipelines
- Data Governance tools such as Purview
Full description
Job Summary
We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.
Job Description: Azure / Databricks Data Engineer (9-15 Years Experience)
Senior Azure Databricks Data Engineer
Experience
9-15 Years of IT Experience
Location
Hybrid
Job Summary
We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.
Key Responsibilities
- Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
- Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
- Develop data models and implement data quality, validation, and governance frameworks.
- Integrate data from multiple sources into a unified Lakehouse architecture.
- Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
- Implement security controls, access management, and data governance using Unity Catalog.
- Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
- Monitor, troubleshoot, and resolve data pipeline issues.
- Support CI/CD, DevOps, and infrastructure automation practices.
- Maintain technical documentation and best practices.
Required Technical Skills Core Technologies
- Databricks Lakehouse Platform
- Apache Spark / PySpark
- Delta Lake
- SQL
- Python
Data Engineering
- ETL / ELT Development
- Data Modeling
- Data Warehousing
- Data Quality & Validation
- Streaming & Real-Time Processing
Governance & Security
- Unity Catalog
- Data Lineage
- Row-Level Security
- Access Control & Compliance