Senior Data Engineer: AI Data Lakehouse Architect - Remote
Senior Data Engineer architecting the enterprise data foundation for a large-scale AI transformation programme in supply chain/procurement. Day to day: build SAP S/4HANA-to-lakehouse ingestion pipelines, RAG-ready unstructured data pipelines, vector stores, and governance/security controls using Python, Spark, Kafka, Airflow, Databricks, and Docker/Kubernetes.
- Compensation
- Not disclosed
- Location
- Abu Dhabi, United Arab Emirates
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Contract
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Expert proficiency in Python, SQL, and modern data engineering frameworks (Apache Spark, Kafka, Airflow)
- Strong experience extracting data from complex ERP environments, specifically SAP S/4HANA and SAP Ariba
- Deep understanding of Data Lakehouse architectures (Databricks/Delta Lake), Relational Databases (PostgreSQL), and Vector Databases (Weaviate/Milvus)
- Experience building pipelines for RAG solutions, conversational agents, and classical ML models using tools such as dbt, Dagster, or Prefect
- Proficiency with containerization (Docker, Kubernetes) and CI/CD pipelines for secure environments
- 5+ years of experience in Data Engineering
- At least 2 years building pipelines for Machine Learning or Generative AI applications in an enterprise setting
- Ability to navigate governance conflicts between agile data work and rigid systems engineering requirements
- Proven collaboration with Data Scientists and Backend Engineers to define data schemas
Nice to have
- Familiarity with SAP BTP
- Experience in Supply Chain, Manufacturing, or Defence sectors
- Ability to understand Bill of Materials (BOM) structures and procurement lifecycles
Full description
Senior Data Engineer
Location: Abu Dhabi, UAE - Remote Global
Type: Contract
Role Summary
We are seeking a Senior Data Engineer to architect and build the foundational data infrastructure for
a large-scale AI transformation programme. You will move beyond standard ETL tasks to design and
deploy enterprise-grade Data Lakehouses that serve as the \"Single Source of Truth\" for AI agents and
advanced analytics platforms.
You will lead the technical execution of data and platform foundation initiatives. Your mandate is to
map rigid enterprise systems (e.g., SAP S/4HANA, Ariba, etc.) and collaborate with teams to
understand complex unstructured data (technical drawings, regulatory text) while building high
performance pipelines and systems that power Intelligent Supply Chain forecasting and generative AI
tools.
You will operate within a structured \"Sprint Zero\" environment, ensuring data lineage, governance,
and security meet strict enterprise standards.
Key Responsibilities
- Data Lakehouse Architecture (Supply Chain)
- ERP Integration: Architect and deploy ingestion pipelines to extract high-volume
transactional data from SAP S/4HANA, Ariba, and PLM systems, ensuring near real-time
availability for forecasting models.
- External Feed Integration: Build connectors for external market intelligence feeds (e.g., S&P
Global, Orbis, EcoVadis) to enrich internal procurement data with macroeconomic and
geopolitical signals.
- Unified Data Model: Design and implement a standardized procurement data model and
taxonomy across multiple entities, harmonizing fragmented datasets into a cohesive layer for
analytics.
- Unstructured Data Pipelines
- Complex Ingestion: Engineer pipelines to ingest and process unstructured technical data,
including PDF tender documents, CAD metadata, and historical CONOPS, transforming them
into vector-ready formats for RAG (Retrieval-Augmented Generation) applications.
- Vector Database Management: Manage and optimize Vector Databases (e.g., Weaviate) to
store embeddings of archival proposals and engineering snippets, ensuring high-speed
retrieval for AI drafting assistants.
- Digital Thread Implementation: Establish data lineage and traceability protocols that link
requirements to physical components, supporting Model-Based Systems Engineering (MBSE)