Senior Data Engineer – Azure Databricks
This Senior Data Engineer will own and maintain a modern data platform on Azure Databricks and Kafka, building end-to-end Spark/Scala data pipelines and dimensional models for a B2B FinTech firm.
- Compensation
- Not disclosed
- Location
- London, United Kingdom
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Commercial experience as a Data Engineer working with Spark in production
- Experience working in finance with financial terminology
- Understanding of software platform architecture
- Good Scala skills (or strong PySpark)
- Strong SQL skills for transformations and data modelling
- Experience building and supporting ingestion pipelines end to end
- Understanding of data modelling for analytics (star schema)
- Exposure to layered data architectures (bronze/silver/gold)
- Experience working in CI/CD and IaC (GitHub Actions, Terraform)
- Track record of writing well-tested data pipelines
- Ability to work as a self-starter with ownership
Nice to have
- Experience with Azure data services (Key Vault, Storage)
- Experience with modern BI tools (Sigma, Looker, Power BI)
- Experience working with AI coding agents
- Interest in analytics and data science
Full description
Senior / Data Engineer - Financial Services / FinTech Platform - Data Infrastructure
Mid-Senior
London, City
Hybrid working - 3 days a week in City office, 2 WFH
Talensa are partnered with a specialist B2B Financial Services Tech platform firm to hire an experienced Data Engineer (Mid-Senior) to take end-to-end ownership of a modern data platform built on Azure Databricks, Kafka and with BI-layer.
This person would be the primary Data Engineer, maintaining and further developing a well-established cloud and CI/CD setup to implement and evolve reliable, well-tested data pipelines in Spark/Scala and to extend the dimensional data models.
You will be part of the technical team collaborating closely with platform engineering, data science, and business stakeholders to further develop the synergy between the internal software platform and the Databricks data platform.
This role will suits a hands-on Data engineer who is looking to work on interesting Tech, solve data problems, while being a key member of the team, comfortable working independently day to day, following agreed designs, patterns, and standards.
What you’ll do
- Own and maintain data pipelines ingesting from various sources into Databricks
- Collaborate closely with software engineers to evolve our software platform and address technical challenges
- Design, develop, and optimize Spark/Scala jobs for ingestion and transformation, using the existing GitHub Actions and Terraform setup for deployments
- Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models
- Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture
- Work closely with analysts, software engineers, and business users using Sigma to ensure they have trusted, well-documented datasets
- Monitor, troubleshoot, and improve pipeline reliability, performance, and cost within the existing observability framework
- Contribute to documentation of pipelines, datasets, and data contracts, acting as the main point of contact for data engineering topics
- Investigate and resolve data quality incidents and pipeline failures
- Proactively drive adoption of Databricks within the team and experiment with new Databricks features and tools where they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows
Tech stack
- Azure Databricks (Spark, Delta Lake)