Senior Data Engineer - AI/Generative AI Platform

About the Role :

We are looking for a Senior Data Engineer / AI Data Platform Engineer with strong expertise in modern data engineering and AI data platforms.

The ideal candidate should have hands-on experience building scalable batch and streaming data pipelines using Python/Scala, Apache Spark, Apache Airflow, and Databricks on AWS, along with strong exposure to RAG architectures and LLM orchestration frameworks.

The role will involve designing modern data lake architectures, integrating enterprise and AI data sources, and enabling GenAI use cases through technologies such as AWS Bedrock, AWS Nova Pro, MCP, and RAG.

Key Responsibilities :Data Engineering & Pipeline Development :

• Design, develop, and maintain scalable data pipelines using Python and/or Scala.

• Build and optimize data processing solutions using Apache Spark.

• Develop and manage workflow orchestration using Apache Airflow.

• Build and manage data pipelines on Databricks running on AWS.

• Develop both batch and streaming data pipelines.

• Implement reliable and scalable data ingestion and transformation processes.

• Work with AWS services including Amazon S3, AWS Lambda, and AWS Glue.

Data Lake & Data Platform Architecture :

• Design and implement modern data lake architectures.

• Work with the Medallion Architecture across : 1. Bronze, 2. Silver, 3. Gold.

• Develop scalable data models and transformation frameworks.

• Ensure data platforms are reliable, maintainable, and optimized for downstream analytics and AI applications.

• Implement appropriate data governance, cataloging, and monitoring practices.

AI Data Platform & GenAI :

• Build data platforms that support GenAI and LLM-based applications.

• Design and implement RAG (Retrieval-Augmented Generation) architectures.

• Work with LLM orchestration frameworks to build AI-enabled data workflows.

• Integrate AI applications with enterprise data sources.

• Work with Model Context Protocol (MCP).

• Develop and integrate solutions using AWS Bedrock.

• Work with AWS Nova Pro for AI/LLM-driven use cases.

• Support data preparation, retrieval, orchestration, and integration requirements for LLM applications.

APIs & Data Integration :

• Design and implement data integration patterns using REST and GraphQL APIs.

• Integrate data from multiple internal and external systems.