Senior Data Engineer
Senior Databricks/Spark Data Engineer at MDW, a Microsoft-focused data, AI and BizApps consultancy (offices in Geneva and Madrid). Day to day you design and deliver enterprise-scale Databricks/Spark data pipelines on Azure and Microsoft Fabric for consulting clients, owning pipelines end to end with a focus on performance tuning, in a hybrid on-site client-facing model in Poland.
- Compensation
- Not disclosed
- Location
- Poland
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Proven experience delivering Databricks projects at very high data volumes, with parallelised execution across countries or markets
- Strong Python and PySpark expertise with deep knowledge of Spark performance tuning and optimisation
- Hands-on experience with Declarative Automation Bundles and CI/CD pipelines (Azure DevOps)
- Experience with end-to-end delivery: from data processing to BI-ready datasets
- Experience with batch data workflow design, including Medallion Architecture
- Experience with data modelling: star schemas, snowflake schemas
- Knowledge of Azure Data Platform: Data Factory, Synapse, Data Lake Gen2, Azure Functions, Logic Apps
- Experience with CI/CD using Azure DevOps
- Knowledge of IaC with Terraform, Bicep, ARM, or Pulumi
- Ability to work with high autonomy and deliver quickly in a fast-paced, evolving project environment
- Strong client communication skills
Nice to have
- Databricks certifications
- Experience working in an Agile environment
- Real-time workflow design (Kafka, structured streaming, Delta Live Tables, Fabric Eventhouse)
- Master Data Management & Governance tools (Profisee, Purview)
- Knowledge of Microsoft Fabric platform
Full description
MDW is a leading consulting company specialised in Data, AI and BizApps, with offices in Geneva and Madrid. Recognised as Microsoft Partner of the Year with multiple awards, we help major organisations transform their business with Azure, Microsoft Fabric, Databricks and AI technologies.
Your role
We are looking for a Senior Databricks / Spark Data Engineer to join our team and work on cutting-edge, enterprise-scale Data projects. Given the criticality of the production phase we are entering, we need deep expertise in Databricks and the big data ecosystem, a proven ability to deliver at very high data volumes, and the seniority to tackle complex challenges autonomously and with confidence.
What you will do
- Design and deliver enterprise-scale Databricks and Spark data solutions, handling very high data volumes with parallelised execution across countries and markets
- Build and evolve Data Solutions on Azure & Microsoft Fabric
- Take end-to-end ownership of data pipelines, from data processing to BI-ready datasets, with a strong focus on performance tuning and scalability
- Deliver real consulting projects across industries, reacting quickly to evolving requirements
- Provide personalised support to clients, working closely with them on-site in a hybrid model, ensuring effective collaboration and solutions that align with their business objectives
- Strengthen seniority, autonomy, and delivery confidence within the team
- Collaborate with an international team of experts at MDW to grow together
What we valueCore Requirements – Databricks & Big Data
- Proven experience delivering Databricks projects at very high data volumes, with parallelised execution across countries or markets
- Strong Python and PySpark expertise, with deep knowledge of Spark performance tuning and optimisation
- Hands-on experience with Declarative Automation Bundles and CI/CD pipelines (Azure DevOps)
- Experience with end-to-end delivery: from data processing to BI-ready datasets
- Experience with Batch Data workflow design, including Medallion Architecture
- Experience with Data Modelling: Star schemas, Snowflake schemas
- Knowledge of Azure Data Platform: Data Factory, Synapse, Data Lake Gen2, Azure Functions, Logic Apps, etc.
Infrastructure & DevOps
- Experience with CI/CD with Azure DevOps
- Knowledge of IaC with Terraform, Bicep, ARM, or Pulumi
Soft Skills & Language
- Ability to work with high autonomy and deliver quickly in a fast-paced, evolving project environment
- Strong client communication skills, ensuring effective management and a positive experience throughout the project