#EG Senior AI Data Engineer
Senior AI Data Engineer at NCS in Singapore who designs an agentic AI harness and multi-agent framework operating a modern data lakehouse built on Apache Iceberg, Trino, and Parquet. Core stack is Python (FastAPI, Pydantic, Pandas) with LangChain/LangGraph, plus data ingestion, transformation, governance, and observability pipelines.
- Compensation
- Not disclosed
- Location
- Singapore, Singapore
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Experience with Generative AI, Agentic AI, or LLM-based solutions
- Proficiency in Python, including FastAPI, Pydantic and Pandas
- Experience designing and implementing AI Harness or multi-agent frameworks
- Hands-on experience with LangChain & LangGraph
- Experience with Apache Iceberg, Trino, and Parquet-based lakehouse architectures
- Experience building data ingestion, transformation, and orchestration pipelines
- Strong knowledge of metadata management, data lineage, and data governance practices
- Experience developing REST APIs, SDKs, and reusable engineering frameworks
- Experience with observability, monitoring, tracing, and operational support practices
- Strong understanding of distributed systems, event-driven architectures, and microservices patterns
- Proven ability to design, develop, and operationalize enterprise-grade AI agents for data discovery, ingestion, transformation, optimization, governance, and observability
- Familiar with modern data engineering technologies, including DuckDB, dbt, and related tools for data transformation, modelling, and pipeline development
- Senior engineers: demonstrated evidence of leading technical designs, guiding junior engineers, working independently with clients and delivery teams, and translating AI/data requirements into scalabl
Full description
NCS is a leading technology services firm that operates across the Asia Pacific region in over 20 cities, providing consulting, digital services, technology solutions, and more. We believe in harnessing the power of technology to achieve extraordinary things, creating lasting value and impact for our communities, partners, and people. Our diverse workforce of 15,000 has delivered large-scale, mission-critical, and multi-platform projects for governments and enterprises in Singapore and the APAC region.
Job Description
Role Summary
You are an autonomous AI Engineer, responsible for creating the harness for designing and building the AI harness, agent framework, and reusable engineering components required to operate a modern data platform based on Apache Iceberg, Trino, and Parquet.
Your primary responsibility is not only to build data pipelines, but also to establish the agentic execution framework (AI Harness) that enables multiple specialized agents to collaborate, execute tasks, maintain state, orchestrate workflows, and deliver end-to-end data engineering capabilities.
Depending on your level of experience, your focus in the team will be on either task execution, independent delivery, or technical leadership.
**Key Responsibilities
AI Harness Development**
Design and implement the foundational agent framework that provides:
- Agent orchestration and coordination
- Workflow execution and state management
- Memory and context handling
- Tool and function calling
- Agent-to-agent communication
- Logging and audit trails
- Error handling and recovery
- Security and access controls
- Observability and monitoring
The harness must support autonomous and multi-agent execution patterns.
Specialized Agents Creation
The agents will need to perform the following:
- Discover Iceberg catalogs and schemas
- Generate metadata and lineage
- Ingest data from databases, APIs, files, and streams
- Register and maintain Iceberg tables
- Generate Trino SQL
- Build transformation workflows
- Create curated and business-ready datasets
- Validate schemas and data integrity
- Detect anomalies
- Analyze Trino workloads
- Recommend partitioning strategies
- Optimize Iceberg tables and Parquet files
- Classify sensitive information
- Manage metadata and lineage
- Apply retention and compliance policies