Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Active TS/SCI required. U.S. citizenship required.

Location: On-site in McLean, VA (Full-Time).

Prove which LLMs actually work for the mission. You'll design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships AI systems they can trust.

What you'll do

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)

• Design and run experiments that benchmark LLMs against defined measures

• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases

• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)

• Use deep learning frameworks such as PyTorch, TensorFlow or Keras

• Leverage GPUs for accelerated training and evaluation

• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets

• Work in GitHub and Jenkins-style CI for reproducible experiments

• Communicate methodological choices and model results to technical and non-technical stakeholders

What you bring (required)

• Active TS/SCI

• Hands-on RAG testing experience (must-have)

• Tuning neural nets / LLMs on custom datasets for real use cases

• Building AI/ML models and ensembles in Python or R

• PyTorch, TensorFlow or Keras (key)

• GPU-accelerated computing (key)

• Testing, evaluating or validating LLM-based AI systems (key)

• Experiment design to benchmark LLMs (key)

• SQL including CTEs, set ops, aggregates and nested subqueries

• GitHub and Jenkins (or equivalent) version control / CI

Nice to have

• Hugging Face Transformers library and hub

• Building benchmarks that measure LLM performance

• Cloud computing development and architecture

• Flask or similar front-end/web frameworks

• Text classification and topic modeling (scikit-learn or deep learning)

• Semantic search applications

• Tableau dashboards

• Verification/validation test benches, Explainable AI (XAI), ONNX

Why Falls

A 9% retirement contribution, 100% of family medical premiums paid, and 4 weeks of PTO. We're a small business that's been delivering since 2019, so your work is visible and your voice counts.