Senior AI Data Engineer

Project Description:
We are looking for AI Engineers for a strategic initiative to build Agentic AI use cases around Trading Surveillance. Ideal candidate is someone with strong pyspark background coupled with proven experience build Agentic AI pipelines for code conversion from Q KDB to Pyspark.
Responsibilities:
Design, develop, and maintain full-stack Python applications with modern frontend frameworks
Build and optimize RAG (Retrieval-Augmented Generation) systems for AI applications
Create and implement efficient vector databases and knowledge stores
Develop APIs that connect frontend interfaces with backend AI services
Implement and maintain CI/CD pipelines for AI applications
Monitor application performance and troubleshoot issues in production
Design and maintain data pipelines, ETL/ELT workflows, and data infrastructure to support AI and analytics workloads
Mandatory Skills Description:
Minimum of 5 years of experience in AI/ML solution implementation (mandatory)
Minimum of 3 years of experience in Pyspark processing high volume data (mandatory)
AI & ML Skills:
Multi-Agent System Design & Orchestration
Full-stack AI Integration (Frontend to Backend)
Advanced Prompt Engineering
LLM Observability, Evaluation & Monitoring (LLMOps)
Data Engineering Skills:
Proficient in Pyspark coding and concepts. Example: Lazy evaluation, window functions, spark architecture and performance finetuning (ex: data skew reduction (salting etc), joins (broadcast, merge sort joins ect)
SQL proficiency and query optimization
Main Technologies:
Pyspark
Git hub Copilot