Databricks Data Engineer
Senior Databricks data engineer (B2B contract, 6 months, hybrid in Cluj) building a Lakehouse platform: Medallion-architecture pipelines with Delta Lake, Airflow orchestration, dbt transformations, and Unity Catalog governance across AWS, Azure, and GCP.
- Compensation
- Not disclosed
- Location
- Cluj, Romania
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Contract
- Experience
- 5+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 5+ years of hands-on data engineering experience, including extensive production experience with Databricks
- Strong expertise in Apache Airflow for workflow scheduling, dependency management, and error handling
- Advanced knowledge of dbt modeling, testing practices, and macro development
- Practical experience with Spec-Driven Development (SDD) and metadata-driven data platform design
- Expert-level proficiency in Python (PySpark) and SQL
- Familiarity with Git, CI/CD processes, and infrastructure-as-code practices
- AWS: configure integrations with Amazon S3, IAM roles, and VPCs
- Azure: implement secure storage with ADLS Gen2, integrate with Microsoft Entra ID, deliver to Power BI
- GCP: build data workflows with Google Cloud Storage, integrate with BigQuery and Vertex AI
Full description
Location: Cluj, Romania Work model:Hybrid, 3 days on site
Contract: B2B
Start: ASAP / 1 month's notice
Duration: 6 months - possibility of extension
Introduction & Summary
Seeking a Senior Databricks Data Engineer to design, build, and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. In this role, you will translate architectural requirements into reliable production solutions, ensuring data contracts, transformations, and orchestration processes are clearly defined, thoroughly tested, and fully automated.
Main Responsibilities
- Design and develop scalable Databricks data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake.
- Lead the adoption of Spec-Driven Development (SDD) by creating machine-readable specifications and data contracts before implementation.
- Build, monitor, and optimize workflow orchestration pipelines using Apache Airflow.
- Develop, test, and maintain modular data transformation models with dbt, integrated with the Databricks SQL and Spark ecosystem.
- Optimize PySpark workloads, manage cluster configurations, and control compute and storage costs.
- Implement data governance, granular access controls, and data lineage tracking through Unity Catalog.
Key Requirements
- 5+ years of hands-on data engineering experience, including extensive production experience with Databricks.
- Strong expertise in Apache Airflow for workflow scheduling, dependency management, and error handling.
- Advanced knowledge of dbt modeling, testing practices, and macro development.
- Practical experience with Spec-Driven Development (SDD) and metadata-driven data platform design.
- Expert-level proficiency in Python (PySpark) and SQL.
- Familiarity with Git, CI/CD processes, and infrastructure-as-code practices.
Multi-Cloud Experience
- AWS: Configure and manage integrations with Amazon S3, IAM roles, and VPCs to support secure and scalable data platforms.
- Azure: Implement secure data storage using Azure Data Lake Storage Gen2 (ADLS Gen2), integrate with Microsoft Entra ID, and deliver data to analytics platforms such as Power BI.
- GCP: Build and operate data workflows using Google Cloud Storage (GCS) while enabling integration with BigQuery and Vertex AI for downstream analytics and AI use cases.