Senior AI Data Engineer

Overview

We are seeking an experienced Senior AI Data Engineer to join a high-impact team delivering innovative data and AI solutions within a complex enterprise environment. This role will be responsible for designing and implementing modern data platforms, AI workflows, and Generative AI solutions using Databricks and Azure technologies.

The successful candidate will combine strong data engineering expertise with hands-on experience building production-grade AI solutions, including Retrieval Augmented Generation (RAG) pipelines, multi-agent workflows, and MLOps capabilities.

Key Responsibilities

• Design and build scalable ETL/ELT and AI pipelines using Databricks, PySpark, Delta Lake and SQL.
• Develop and maintain Lakehouse architectures using Bronze, Silver and Gold data layers.
• Build and optimise Delta Live Tables, Databricks Workflows and job orchestration processes.
• Develop and deploy production-grade Generative AI and agent-based solutions.
• Create and maintain RAG pipelines and AI agent frameworks.
• Support MLOps activities, including model training and inference pipelines using MLflow.
• Implement data quality monitoring, governance controls and metadata management through Unity Catalog.
• Develop event-driven integrations using Azure cloud services.
• Collaborate with cross-functional teams to create feature stores and AI training datasets.
• Integrate Databricks with wider Azure services including ADLS Gen2, Azure Data Factory, Azure Key Vault and Azure DevOps.
• Contribute to the development of data engineering best practices and centre of excellence initiatives.
• Mentor and support engineering teams in AI technologies and capabilities.

Essential Skills & Experience

• Strong experience as a Data Engineer, AI Engineer, Machine Learning Engineer, or similar role.
• Hands-on experience building and deploying production-grade:
• Multi-agent AI workflows
• Retrieval Augmented Generation (RAG) solutions
• AI agent frameworks
• Data pipeline solutions within Databricks

• Strong programming skills in Python and SQL.
• Experience with Apache Spark and distributed data processing.
• Experience implementing Generative AI solutions within Databricks Workflows and Jobs.
• Strong knowledge of Databricks Lakehouse architecture.
• Hands-on experience with:
• Delta Lake
• Unity Catalog
• Databricks Workflows
• MLflow

• Experience implementing event-driven integrations using:
• Azure Service Bus (essential)
• Azure Functions (essential)