Research Scientist, LLM Evaluation & Post-Training
- Compensation
- $150K–$300K / yr
- Location
- Remote Work( USA)
- Work setup
- Remote
- Seniority
- Not specified
- Employment
- Full time
- Experience
- Not specified
- Category
- Ai engineering
- Posted
Skills
Must have
- Education: MS or PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative field (PhD strongly preferred).
- Research Experience: 5+ years of relevant experience in applied ML research or research science, with substantial work in LLMs or foundation models (graduate research counts).
- LLM Evaluation Expertise: Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
- Experimental Design: Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems.
- Technical Proficiency: Strong Python coding skills for research experimentation, data processing, evaluation pipelines, statistical analysis, and visualization. Hands-on experience with modern ML fram
- Evaluation Methodology: Ability to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability. Experience designing reproducible e
- Communication: Strong written and verbal communication skills; able to present nuanced technical conclusions, assumptions, and limitations clearly to both research and non-technical audiences.
Nice to have
- Post-Training Practice: Hands-on experience running fine-tuning or post-training experiments (SFT, preference optimization, RLHF/RLAIF-style workflows).
- Multimodal & Long-Context: Experience with multimodal evaluation (text-image, audio, video) and long-context benchmarking in real-world settings.
- Agentic Evaluation: Experience designing multi-turn, interactive, or agentic evaluation protocols.
- Scientific Contribution: Publications and/or open-source benchmark contributions in LLM evaluation, post-training, alignment, or related areas at top venues (NeurIPS, ICML, ICLR, ACL, EMNLP, etc.).
- Applied Research Consulting: Experience in customer-facing applied research, technical consulting, or cross-functional product/research collaboration.
- Safety & Governance: Familiarity with safety, trustworthiness, and governance considerations in GenAI evaluation.
Full description
About Centific
Centific is a frontier AI data foundry that curates diverse, high-quality data, using our purpose-built technology platforms to empower the Magnificent Seven and our enterprise clients with safe, scalable AI deployment. Our team includes more than 150 PhDs and data scientists, along with more than 4,000 AI practitioners and engineers. We harness the power of an integrated solution ecosystem—comprising industry-leading partnerships and 1.8 million vertical domain experts in more than 230 markets—to create contextual, multilingual, pre-trained datasets; fine-tuned, industry-specific LLMs; and RAG pipelines supported by vector databases. Our zero-distance innovation™ solutions for GenAI can reduce GenAI costs by up to 80% and bring solutions to market 50% faster.
Our mission is to bridge the gap between AI creators and industry leaders by bringing best practices in GenAI to unicorn innovators and enterprise customers. We aim to help these organizations unlock significant business value by deploying GenAI at scale, helping to ensure they stay at the forefront of technological advancement and maintain a competitive edge in their respective markets.
About JobResearch Scientist, LLM Evaluation & Post-TrainingCompany: CentificLocation: Palo Alto, CA or Seattle, WA (Hybrid/Remote)Type: Full-time
##
Role Overview
As a Research Scientist, LLM Evaluation & Post-Training, you will be at the frontier of how evaluation design, measurement strategy, and feedback signals drive model improvement across Centific’s AI platform products. This is a high-impact individual contributor and collaborative research role that sits at the intersection of applied ML research, enterprise AI product development, and customer-facing scientific consulting.
You will lead research programs that define next-generation evaluation-driven post-training workflows, develop rigorous benchmark frameworks, and partner directly with leading AI organizations to deliver credible, actionable model improvement insights. This role offers the opportunity to shape Centific’s internal research agenda, build reusable scientific assets, and publish at top-tier venues.
Key Responsibilities
- Research Agenda & Experimentation: Define and execute a rigorous research agenda focused on LLM evaluation and post-training, with emphasis on evaluation-driven model improvement. Design experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes.
- Evaluation Framework Development: Develop and validate comprehensive evaluation frameworks for LLM and multimodal systems, covering benchmark and task design, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness/stress testing.
- Advanced Evaluation Research: Lead research on frontier evaluation domains including long-context, cross-modal, and dynamic multi-turn evaluations. Study effectiveness and limitations of existing techniques and propose improved methodologies with clear validity and scalability tradeoffs.