Senior Data Engineer
Senior Data Engineer building a configuration-driven data platform using Databricks and Snowflake to support regulatory reporting for financial institutions. The role involves migrating legacy Python/MySQL systems to a modern lakehouse architecture, designing auditable PySpark pipelines, and providing technical leadership.
- Compensation
- Not disclosed
- Location
- London, United Kingdom
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 5+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 8+ years as a software engineer
- 5+ years in data engineering
- Experience designing auditable, reproducible data pipelines in regulated environments
- Proficiency in Python and PySpark
- Able to write and optimize complex SQL queries on large data sets
- Strong data modelling and warehouse design fundamentals
- Software engineering fundamentals (clean code, DRY, SOLID, automated testing, CI/CD)
- Deep experience with cloud lakehouse/data warehouse (Databricks, Snowflake, etc.)
- Experience with a workflow orchestrator (e.g. Airflow, Dagster, or Databricks Workflows)
- Proven system and platform design judgment
- Comfortable translating complex regulatory requirements into technical specifications
- Able to own large, ambiguous initiatives and deliver incrementally
- Strong communication and collaboration skills
Nice to have
- Experience with event-driven architecture patterns
- Hands-on experience with AWS cloud infrastructure
- Practical experience with infrastructure as code (e.g. Terraform)
- Experience modernizing legacy systems
- FinTech, RegTech, or financial services background
Full description
Job Overview:
As a Senior Data Engineer, you will be developing our configuration-driven data platform in Databricks, enabling non-engineers to define regulatory logic, and our Snowflake data warehouse, ensuring scalability, auditability, and fitness for client-facing regulatory use cases. You’ll be writing clean, maintainable, and well-tested code that follows best practices. As a senior member of the team, you will be providing technical leadership and mentorship to your colleagues.
You’ll join at a pivotal stage as we modernize our data infrastructure, migrating from Python scripts and MySQL to Databricks and Snowflake. Our systems power regulatory reporting for major financial institutions, requiring precision, traceability, and reliability.
Key Responsibilities
- Design and evolve our configuration-driven data framework in Databricks and our Snowflake warehouse.
- Coach and mentor junior engineers, and lead technical discussions and design reviews.
- Design, build, and optimize data pipelines using Databricks, Python and PySpark, with event-driven components (e.g. Kafka).
- Ensure pipelines are auditable, lineage-aware, idempotent, and resilient in regulated environments.
- Implement robust data quality controls including testing, validation, monitoring, and alerting.
- Drive performance optimization across Spark and Snowflake workloads.
- Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs.
- Contribute to the continuous improvement of our development processes and tools.
About You:
Must-Haves:
- 8+ years as a software engineer, including 5+ years in data engineering, owning system design and delivery.
- Experience designing auditable, reproducible data pipelines in regulated or high-integrity environments using Python and PySpark.
- Able to write and optimize complex SQL queries on large data sets.
- Strong data modelling and warehouse design fundamentals.
- Strong software engineering fundamentals: clean code, DRY, SOLID, automated testing, CI/CD, and observability, with a bias toward simple, maintainable solutions.
- Deep experience with a cloud lakehouse / data warehouse (Databricks, Snowflake, BigQuery, Redshift, EMR, or similar). Our stack is Databricks and Snowflake; strong equivalent experience elsewhere transfers.
- Experience with a workflow orchestrator (e.g. Airflow, Dagster, or Databricks Workflows).