Senior Data Engineer: Databricks & Delta
Contract role designing scalable data pipelines on Databricks and Delta Lake for an energy trading company in London — building real-time streaming analytics, ingesting SAP data via Qlik Replicate and ADF, and optimizing data models on Azure for reporting and decision-making.
- Compensation
- Not disclosed
- Location
- London, United Kingdom
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Strong experience with Databricks — DLT, Delta Lake concepts, UC governance
- Solid understanding of streaming technologies (Spark Structured Streaming, Autoloader)
- Programming skills in Python, SQL, or Scala
- Proficiency in data modelling, ETL/ELT processes, and data architecture
- Experience in MS Copilot, AI Foundry, Databricks (MosaicAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem-solving skills
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
- Hands-on expertise in ADF and Qlik Replicate for data ingestion and replication
- Experience working in Azure cloud environments
- Collaboration with cross-functional teams
- Attention to data quality and governance
Full description
Overview
In this role you will design and implement scalable data pipelines and enable real-time and batch analytics for a leading energy trading company. You will work with cross-functional teams to build robust data solutions that support decision-making and reporting at scale. The position focuses on Databricks, Delta Lake, streaming architectures, and SAP data ingestion, offering an opportunity to impact data-driven decisions across the organization. This contract role combines hands-on engineering with shaping data platforms to support energy trading operations.
Responsibilities
- Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
- Develop and manage streaming data solutions for real-time analytics
- Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality
- Perform data modelling and optimize data structures for analytics and reporting
- Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
- Monitor, troubleshoot, and enhance data workflows for performance and reliability
- Maintain technical documentation and contribute to data engineering best practices
Key requirements
- Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
- Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
- Programming skills in Python, SQL, or Scala
- Proficiency in data modelling, ETL/ELT processes, and data architecture
- Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem-solving skills
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
- Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
- Experience working in Azure cloud environments
- collaboration with cross-functional teams
- problem-solving and analytical thinking
- attention to data quality and governance
- Databricks (DLT, Delta Lake)
- Streaming technologies (Spark Structured Streaming, Autoloader)
- Python, SQL, or Scala