Staff GPU Inference SDET
Founding quality/validation engineer (SDET) for Cerebras's new GPU inference team, designing automated test, benchmarking, fault-injection, and release-qualification pipelines for multi-node GPU clusters and LLM serving stacks. Core tech: Python, Kubernetes/Slurm/Ray, NVIDIA/AMD GPU stacks, Prometheus/Grafana, and distributed LLM serving frameworks.
- Compensation
- Not disclosed
- Location
- Sunnyvale, CA
- Work setup
- Hybrid
- Seniority
- Staff
- Employment
- Not specified
- Experience
- 8+ years
- Category
- Qa
- Posted
Skills
Must have
- 8+ years of software engineering experience as an SDET, Infrastructure Quality Lead, or Systems Test Engineer
- Hands-on experience bringing up, provisioning, and validating multi-node GPU clusters (NVIDIA or AMD) across cloud or enterprise data center environments
- Deep understanding of LLM serving engines and distributed runtimes, including prefill vs. decode disaggregation, KV-cache management, and dynamic batching
- Expert-level Python programming with experience designing custom test automation frameworks, diagnostic tooling, and CI/CD integration
- Strong proficiency with container orchestration tools (Kubernetes, Slurm, Ray) and high-performance cluster interconnects (InfiniBand, RoCE, NCCL)
- Proven background in root-cause analysis across software/hardware boundaries, stress testing, and node failure simulation in distributed systems
Nice to have
- Direct experience with either AMD (ROCm / HIP) or NVIDIA software stacks
- Experience building workload replay tools, ML evaluation pipelines, or MLPerf Inference benchmark suites
- Familiarity with low-level kernel profiling tools (PyTorch Profiler, NVTX, ROCm profilers) or C++
Full description
Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. This architecture allows Cerebras to deliver industry-leading training and inference speeds; over 10 times faster than GPU-based hyperscale cloud inference services.
This order of magnitude increase in speed is transforming the user experience of AI applications, unlocking real-time iteration and increasing intelligence via additional agentic computation.
Cerebras works with the leading model labs, global enterprises, and cutting-edge AI-native startups. OpenAI recently announced a multi-year partnership with Cerebras, to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference.
About the Role
As a Staff GPU Inference SDET, you will be the founding quality, reliability, and validation lead for a new GPU Inference Development team. Working closely with engineering leads and cross-functional systems infrastructure teams, you will design, build, and scale the end-to-end release qualification and automated test ecosystem for our GPU inference stack and rack-scale accelerated compute fleets. In this high-impact role, you will be responsible for building automated test suites to validate multi-node GPU cluster bring-up, verifying prefill worker optimizations, testing open-source and custom serving engines, and ensuring numerical correctness and performance stability under real-world streaming workloads. You will be the primary technical anchor ensuring production-grade reliability, fault isolation, and peak inference performance across accelerated GPU infrastructure.
WHAT YOU’LL DO
Build GPU Release Qualification Systems:
Design and implement automated test automation frameworks, regression gates, and release qualification pipelines for the complete GPU inference stack—spanning custom API services, model-serving workers, container runtimes, serving engines, driver stacks, and firmware.
Inference Serving & Workload Validation:
Benchmark and stress-test distributed LLM serving frameworks, focusing on prefill vs. decode worker performance, continuous batching, prefix caching, KV-cache efficiency, and tensor/expert parallelism.
Performance & Performance Modeling Verification:
Build automated workload replay and benchmarking tools to validate GPU performance models. Track critical serving metrics including Time-to-First-Token (TTFT), Inter-Token Latency (ITL), request throughput, tail latency (P99), and capacity efficiency.
Numerical Correctness & Quality Gates:
Build validation infrastructure to ensure model accuracy, precision stability (FP16/FP8/quantization), determinism, and output correctness across software updates, kernel fusions, and hardware revisions.