CDI Data Engineer
- Compensation
- Not disclosed
- Location
- Ascot
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Full time
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Strong hands-on background in Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting
- Workflow orchestration and dependency management Idempotency, retry/restart and failure-recovery patterns CI/CD, version control and automated deployment
- Worked across monitoring, logging and operational alerting
- Completed data-quality and reconciliation controls API/file/SFTP integration patterns
Full description
3 CDI Data Engineering vacancies, differing mainly in where they sit within the customer data lifecycle.
The first is the most data-centric, the second is the most DevOps/MLOps-oriented, and the third is the most transformation-focused.
2-3 days a week in office based in Canary Wharf.
Client Details
Our client operates within the Oil and Gas sector. British Petroleum.
Description
Role 1: CDI Data Engineer
- Implement and maintain SCI base, fact and curated pipelines.
- Deliver Elavon/PAR/WRE/payment and loyalty matching.
- Support ICE-to-EV data engineering and dashboard datasets.
- Implement DO-site and market-specific logic.
- Investigate volume drops, match-rate changes and data completeness defects.
- Execute controlled reloads, production validation, DQ/freshness controls and BAU support.
- Prepare PRs, tests, technical designs, CAB/release evidence and stakeholder
Role 2: BNA Production-Readiness, MLOps & Integration Engineer:
- Automate end-to-end BNA orchestration
- Reliable retry/recovery and idempotent processing
- Production monitoring and alerting automated activation and acknowledgement handling
- Reducing manual intervention Stable QA production deployment process
- Runbooks and operational handover
- Reducing runtime and production failures
Role 3: CIS/SCI Integration & Post-Migration Stabilisation Engineer:
- Repointing remaining CIS/SCI dependencies
- Stabilising migrated pipelines
- Reusable error-handling patterns
- Reducing legacy technical debt
- Removing dual-platform monitoring
- Resolving SFMC/Comarch integration issues
- Reducing production support burden
- Clear handover into BAU
Profile
Role 1: CDI Data Engineer
Must Have worked with:
- Strong SQL, Python/Spark and Databricks pipeline engineering background.
- Large-scale transaction, payment, loyalty, EPOS/WRE and identity-matching work history
- Historical/full/incremental loads, high-water marks and reconciliation.
- Azure DevOps PRs, unit testing, CI/CD and production release support.
Nice to have: Reporting/Power BI dataset exposure desirable. SFMC/CIS integration and multi-market SCI knowledge desirable.
Role 2: BNA Production-Readiness, MLOps & Integration Engineer
Must Have:
- Strong hands-on background in Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting