Data Engineer (Databricks)
A 6-month contract (inside IR35) Data Engineer role, mostly remote from Leeds, running and maintaining production data pipelines on Databricks and AWS for a large-scale UK Government project. It blends L2/L3 production support (incident triage, RCA, fixes) with hands-on data engineering using Databricks, PySpark, SQL, Python, Delta Lake and AWS services like S3, IAM and CloudWatch.
- Compensation
- Not disclosed
- Location
- Rodley, United Kingdom
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Strong hands-on Databricks experience
- PySpark experience
- SQL experience
- Python experience
- Hands-on AWS experience, particularly S3, IAM and CloudWatch
- Databricks: notebooks, scheduled jobs, cluster configuration, driver/executor logs, Spark UI and performance troubleshooting
- Supporting and maintaining production data pipelines: incident investigation, safe recovery, root cause analysis and permanent remediation
- Experience with Spark SQL, Delta Lake, Parquet and Hive metastore tables, including schemas, partitions and table metadata vs underlying files
- Understanding of role-based access, KMS encryption and diagnosing data access failures
- Owning incidents end-to-end and diagnosing issues across Python, PySpark, SQL, Databricks jobs and AWS integrations
- Working knowledge of Git, code reviews, CI/CD and controlled production deployments incl. testing, rollback and release validation
Nice to have
- AWS Glue
- Lambda
- Step Functions
- Linux
- Shell Scripts
- Terraform
- Gitlab
- Jenkins
Full description
Data Engineer (Databricks)
Leeds (Hybrid)
6 Month Contract
£(Apply online only)/day (Inside IR35)
Hybrid Working Model - primarily working from home with travel to the Leeds office 1-2 days per month.
Start ASAP in October 2026.
Data Engineer needed with strong hands‑on Databricks, PySpark, AWS, SQL + Python experience.
A chance to work with a leading global IT transformation on a large‑scale Government project:
- Hands‑on Data Engineer to run + maintain production data pipelines on Databricks and AWS.
- Keeping scheduled data processing reliable, resolve incidents, fixes + improving data services performance.
- The role combines L2/L3 production support and Data Engineering tasks.
- Databricks experience covering notebooks, scheduled jobs, cluster configuration, driver / executor logs, Spark UI + performance troubleshooting.
- Supporting + maintaining production data pipelines, including incident investigation, safe recovery, root cause analysis + permanent remediation.
- Experience of Spark SQL, Delta Lake, Parquet + Hive metastore tables, including schemas, partitions and the relationship between table metadata and underlying files.
- Hands‑on AWS experience, particularly S3, IAM and CloudWatch, with an understanding of role‑based access, KMS encryption and diagnosing data access failures.
- Owning incidents from investigation through recovery and closure. Diagnosing issues in Python, PySpark, SQL, Databricks jobs and AWS integrations + providing clear progress updates and timely escalation.
- Working knowledge of Git, code reviews, CI/CD and controlled production deployments, including testing, rollback and release validation.
Advantageous: AWS Glue, Lambda, Step Functions, Linux, Shell Scripts, Terraform, Gitlab, Jenkins