Data Engineer (Databricks)
Six-month hybrid contract (Inside IR35) combining L2/L3 production support with data engineering on Databricks and AWS for a large-scale government project. The engineer keeps scheduled data pipelines reliable — incident investigation, recovery, root cause analysis, and performance tuning — using PySpark, SQL, Python, S3, IAM, and CloudWatch. Based in Leeds, mostly work-from-home with 1–2 office d
- Compensation
- Not disclosed
- Location
- Pride Park, Derby
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Strong hands-on experience with Databricks, PySpark, AWS, SQL, and Python
- Databricks experience with notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI, and performance troubleshooting
- Experience with Spark SQL, Delta Lake, Parquet, and Hive metastore tables, including schemas, partitions, and table metadata vs underlying files
- Hands-on AWS experience, particularly S3, IAM, and CloudWatch; role-based access, KMS encryption, and diagnosing data access failures
- Working knowledge of Git, code reviews, CI/CD, and controlled production deployments, including testing, rollback, and release validation
Nice to have
- Experience with AWS Glue, Lambda, Step Functions, Linux, shell scripts, Terraform, GitLab, or Jenkins
Full description
Salary: £? - ? per yearRequirements:
- Strong hands-on experience with Databricks, PySpark, AWS, SQL, and Python.
- Databricks experience with notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI, and performance troubleshooting.
- Experience with Spark SQL, Delta Lake, Parquet, and Hive metastore tables, including schemas, partitions, and the relationship between table metadata and underlying files.
- Hands-on AWS experience, particularly with S3, IAM, and CloudWatch; understanding of role-based access, KMS encryption, and diagnosing data access failures.
- Working knowledge of Git, code reviews, CI/CD, and controlled production deployments, including testing, rollback, and release validation.
- Advantageous: experience with AWS Glue, Lambda, Step Functions, Linux, shell scripts, Terraform, GitLab, or Jenkins.
Responsibilities:
- Run and maintain production data pipelines on Databricks and AWS.
- Keep scheduled data processing reliable by resolving incidents, applying fixes, and improving data services performance.
- Support and maintain production data pipelines, including incident investigation, safe recovery, root cause analysis, and permanent remediation.
- Own incidents from investigation through recovery and closure; diagnose issues across Python, PySpark, SQL, Databricks jobs, and AWS integrations.
- Provide clear progress updates and escalate issues in a timely manner.
Technologies:
- AWS
- AWS Glue
- Lambda
- CI/CD
- CloudWatch
- Databricks
- Git
- GitLab
- Hive
- IAM
- Support
- Jenkins
- Linux
- Python
- PySpark
- SQL
- Spark
- Terraform
- UX UI Design
- Cloud
More:
We are seeking a Data Engineer for a large-scale government project as part of a global IT transformation. This is a six-month contract based in Leeds, with a hybrid working model: primarily working from home, with travel to our Leeds office one to two days per month. The role combines L2/L3 production support with data engineering work and is due to start as soon as possible in October 2026. The rate is listed as £ per day, Inside IR35.
last updated 44 week of 2026