Data Engineer
- Compensation
- Not disclosed
- Location
- Glasgow, Scotland, gb
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Nice to have
- Experience with Starburst/Trino, AWS S3 concepts, and distributed query behavior for large-dataset validation and troubleshooting.
- Exposure to React JS or API-driven operational user interfaces for configuration, monitoring, and support workflows.
- Familiarity with OpenShift, Kubernetes, Helm, or container platform deployments in enterprise environments.
- Understanding of financial data controls, auditability, reconciliation, data lineage, operational resilience, and data quality expectations.
Full description
Anticipated Contract End Date/Length: December 31, 2026
Work set up: Hybrid (2–3 days per week in office)
Clearance required: BPSS checks. Candidates must be eligible for BPSS clearance
Our client in the Information Technology and Services industry is looking for a Data Engineer with strong Spark, Scala, and Python skills to develop, maintain, and troubleshoot data orchestration workflows and reusable engineering components. The role will support the processing, transformation, validation, and delivery of large-scale financial datasets, while contributing to platform engineering, production support, monitoring, automation, and operational resilience. The successful candidate will work across application, platform, database, infrastructure, and downstream teams to maintain reliable data processes and ensure compliance with technology, security, and change standards.
Responsibilities
- Develop, maintain, and troubleshoot data orchestration workflows using Apache Airflow for extract generation, data publishing, feed delivery, and operational batch processes.
- Build and support reusable engineering components using Python, Scala, and Sharedview Spark to process, transform, and validate large-scale financial datasets.
- Design, write, and optimize SQL/PostgreSQL and Starburst/Trino queries for metadata processing, data validation, reconciliation, and operational investigation.
- Support Output Gateway production issues, including failed DAGs, delayed extracts, missing files, data mismatches, checksum failures, logs, status updates, and downstream delivery exceptions.
- Contribute to platform engineering and release activities using GitLab, Docker, and controlled CI/CD practices across DEV, UAT, and production-like environments.
- Enhance monitoring, logging, alerting, audit tracking, and automation to improve operational stability, supportability, and evidence for control processes.
- Work with application, platform, database, infrastructure, and downstream teams to diagnose issues end-to-end and implement safe, controlled fixes.
- Maintain a strong risk and control mindset by ensuring changes are tested, reviewed, documented, and released in accordance with technology, security, and change standards.
- Strong skills and experience in data engineering using Spark, Scala, and Python.
- Experience developing, maintaining, and troubleshooting Apache Airflow workflows for data orchestration and operational batch processes.
- Hands-on experience building reusable engineering components to process, transform, and validate large-scale financial datasets.
- Proficiency in SQL and PostgreSQL, including query design, optimization, metadata processing, data validation, and reconciliation.