Senior ML Engineer - MLOps and AI Evaluation
ELLKAY, a healthcare data management and interoperability company, is hiring a Senior ML Engineer to build the shared MLOps and AI evaluation foundation for its classical ML, LLM, and agent services. Day to day: reproducible training/deployment pipelines, model/prompt/dataset versioning, CI/CD evaluation gates, benchmarking, telemetry, drift monitoring, and rollback.
- Compensation
- Not disclosed
- Location
- IN
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- Not specified
- Category
- Ml / AI
- Posted
Skills
Must have
- Implement model, prompt, retrieval-index, and evaluation-dataset versioning with lineage and reproducible release evidence
- Build CI/CD evaluation gates for mapping, clinical extraction, recommendations, RAG, and agents
- Validate automated judges against SME labels
- Create synthetic healthcare fixtures and benchmarks for malformed data, corrections, missing evidence, rare cases, and permission boundaries
- Deploy serving pipelines and monitor drift, model quality, latency, failure rates, and unit cost
- Support shadow testing, canaries, rollback, and fallback
- Partner with Security and Data Platform on tenant isolation tests, PHI-safe telemetry, approved training rights, and model-vendor data handling
- Optimize inference resources and cost through profiling, batching, caching, and model sizing
- Own reproducible model-training jobs and dependency tracking
- Build reusable services for ELLKAY products while preserving source evidence, approved data-use rights, and tenant isolation
Full description
ELLKAY started out providing connectivity solutions to laboratories and within a few years, grew to also provide data management solutions to ambulatory organizations. ELLKAY is now a trusted data management partner in five healthcare segments. ELLKAY’s solutions continue to serve laboratories and ambulatory practices and have expanded to empower hospitals and health systems, healthcare IT vendors, ambulatory practices, health plans, and other healthcare organizations with cutting-edge technologies and solutions that drive their growth and interoperability strategies.
Today, ELLKAY remains true to our core values, building strong partner relationships and offering unparalleled service and support while providing innovative, scalable solutions to the challenges our customers face in today’s data-rich world.
ELLKAY's experience, customer-focused approach, and reputation for innovation, speed, and accuracy differentiate ELLKAY as a premier partner for your interoperability needs and data management strategy.
Job Description:Ellkay is looking for an Senior ML Engineer - MLOps and AI Evaluation to build the shared ML operations and evaluation foundation for ELLKAY's classical ML, LLM, and agent services. The engineer will implement reproducible training/deployment pipelines, versioned models and prompts, benchmark execution, synthetic fixtures, release gates, telemetry, and rollback. The role owns the engineering systems that make quality, reliability, and cost visible throughout the AI lifecycle.
Essential Duties & Responsibilities:
- Implement model, prompt, retrieval-index, and evaluation-dataset versioning with lineage and reproducible release evidence.
- Build CI/CD evaluation gates for mapping, clinical extraction, recommendations, RAG, and agents. Validate automated judges against SME labels.
- Create synthetic healthcare fixtures and benchmarks for malformed data, corrections, missing evidence, rare cases, and permission boundaries.
- Deploy serving pipelines and monitor drift, model quality, latency, failure rates, and unit cost. Support shadow testing, canaries, rollback, and fallback.
- Partner with Security and Data Platform on tenant isolation tests, PHI-safe telemetry, approved training rights, and model-vendor data handling.
- Optimize inference resources and cost through profiling, batching, caching, and appropriate model sizing. Own reproducible model-training jobs and dependency tracking.
- Partner with Data Platform and product engineers, clinical/data SMEs, DME/QA, and security teams.
- Build reusable services for ELLKAY products and preserve source evidence, approved data-use rights, and tenant isolation.
- Model or agent proposals follow established human approval and platform release workflows.