Senior Data Engineer/AI Data Platform Engineer - Customer Experience

Role Overview :

We are building a next-generation data platform for a leading Media & Entertainment organization to unify Customer Experience (CX) data across multiple sources-including application reviews, social media platforms, and customer interaction channels.

This platform will ingest, process, and transform large-scale data into a modern data lake architecture, and power an intelligent insights layer using advanced analytics and Generative AI. The goal is to enable real-time, actionable insights that improve customer engagement and business decision-making.

Key Responsibilities :

• Design and build scalable data pipelines to ingest CX data from multiple sources such as app reviews, social media (Twitter, Instagram), and internal systems

• Implement robust ETL/ELT pipelines using Apache Spark and orchestrate workflows using Apache Airflow

• Develop and maintain data lake architecture following the Medallion (Bronze, Silver, Gold) pattern

• Work with Databricks on AWS to build and optimize distributed data processing systems

• Build an intelligent data layer leveraging analytics and Generative AI for actionable insights

• Design and implement RAG (Retrieval-Augmented Generation) pipelines for contextual insights

• Integrate LLM orchestration frameworks and Model Context Protocol (MCP) for scalable AI workflows

• Utilize AWS Bedrock (or equivalent platforms) to deploy and manage foundation models

• Ensure data quality, governance, security, and observability across pipelines

• Collaborate with data scientists, product teams, and business stakeholders to deliver high-impact solutions

Required Skills & Experience :

• Strong experience in Python & PySpark for data engineering (Must)

• Strong experience with Databricks platform for data processing (Must)

• Hands-on experience with Apache Spark for large-scale data processing

• Expertise in building and managing data pipelines using Apache Airflow

• Experience with AWS ecosystem (S3, Lambda, Glue, etc.)

• Solid understanding of data lake architectures and Medallion architecture

• Experience working with streaming and batch data pipelines

• Strong knowledge of REST/GraphQL APIs and data integration patterns

• Familiarity with data governance, cataloging, and monitoring tools

GenAI & Advanced Capabilities :

• Experience with LLM orchestration frameworks

• Strong understanding of RAG (Retrieval-Augmented Generation) architectures