Nscale
Principal Infrastructure Engineer, AI Cluster Performance & Validation
A principal-level engineer on Nscale's AI Infrastructure Operations team who defines how multi-thousand-GPU AI clusters are validated and tuned for production: building Python control-plane and burn-in tooling, running real distributed training/inference workloads, and debugging performance across GPUs, InfiniBand/RoCE fabrics, storage, and SLURM/Kubernetes schedulers.
- Compensation
- Not disclosed
- Location
- Houston; New York; San Francisco; Seattle
- Work setup
- Not specified
- Seniority
- Principal
- Employment
- Not specified
- Experience
- 10+ years
- Category
- Devops
- Posted
Skills
Must have
- Bachelor's or higher degree in Computer Science, Computer Engineering, or related technical field, or equivalent practical experience
- 10+ years building, operating, or debugging large-scale compute infrastructure, including significant time at staff/principal level owning cross-team technical direction
- Hands-on experience running real AI compute jobs at scale (pre-training, fine-tuning, large-scale inference), including distributed training strategies (data, tensor, pipeline, expert parallelism) and
- Experience validating and accepting large clusters (thousands of GPUs) with command of benchmark methodology
- Proven ability to diagnose distributed performance problems using tools such as NCCL debug tracing, Nsight Systems/Compute, PyTorch Profiler, perf, and fabric telemetry
- Deep understanding of high-performance fabrics (InfiniBand and/or RoCEv2, RDMA, GPUDirect, adaptive routing, congestion control, rail-optimized topologies) and networking fundamentals (TCP/IP, BGP)
- Deep Linux systems expertise (kernel tunables, NUMA, PCIe, IRQ, memory behavior) and strong production Python
- Experience with C/C++ or Go and with configuration management tooling (e.g., Ansible, Terraform)
- Experience operating and debugging AI workloads under SLURM and/or Kubernetes at scale
Nice to have
- Master's degree or PhD in Engineering, Computer Science, or related technical field
- Experience bringing up and qualifying a greenfield GPU supercluster from first rack to production traffic, including firmware, driver, and topology standardization
- Direct experience with NVIDIA GPU platforms (H200/GB200/GB300-class), NVLink/NVSwitch, DCGM, SHARP, UFM, and the NVIDIA software stack, or equivalent depth on AMD Instinct and ROCm/RCCL
- Published or presented benchmark, scaling, or post-mortem work (e.g., MLPerf submissions, scaling studies, public technical write-ups)
- Experience with advanced observability systems (Prometheus, Grafana, OpenTelemetry) applied to high-cardinality GPU and fabric telemetry
- Experience with high-throughput parallel storage (Lustre, GPFS, WEKA, VAST) and diagnosing data-pipeline-bound training jobs
- Familiarity with cloud-native technologies (Kubernetes, Docker), infrastructure-as-code, and bare metal/DCIM tooling (NetBox, MAAS, Ironic, IPMI, Redfish)
- Demonstrated ability to integrate AI tools to optimize/redesign workflows with measurable impact
- Familiarity with SLOs/metrics measurement and logs/telemetry/metrics integration
Full description
Overview
As a Principal Infrastructure Engineer, AI Cluster Performance & Validation, you will be a critical member of the AI Infrastructure Operations team, responsible for ensuring the acceptance, performance, and scalability of our cutting-edge AI and High-Performance Computing (HPC) environments. Leveraging software engineering and testing principles, you will focus on building and maintaining the control plane, tooling, and automation that supports performance and validation testing of large-scale AI clusters. Your work will directly translate into higher system availability, compute optimization and reduced operational costs.
Key Responsibilities
- Own the technical definition of "healthy at scale." Set the architecture, roadmap, and acceptance criteria by which multi-thousand-GPU clusters are declared production-ready, and establish the performance bar (collective bandwidth, job goodput, model FLOPs utilization) that every cluster must clear before and after customer handover. Establish technology and product direction in collaboration with other tech leads, managers, and senior leadership.
- Run and instrument real AI workloads as a diagnostic instrument. Stand up and execute distributed training and inference jobs — open-source and customer-representative models — across thousands of accelerators to validate cluster behavior under genuine load rather than synthetic proxies alone, and translate what those runs reveal into fleet-wide fixes.
- Lead deep diagnosis of large-scale cluster failures and performance regressions, isolating root cause across the full stack: GPU and NIC firmware, PCIe/NVLink topology and NUMA placement, InfiniBand/RoCE fabric health, congestion control and routing, storage and data-loader throughput, scheduler placement, and framework/communication-library behavior. Serve as the final escalation point for the hardest slow-job and stalled-job investigations.
- Design and build the validation and burn-in systems that qualify nodes, racks, and full pods at scale — NCCL/RCCL collective sweeps, HPL/HPCG and MLPerf-style benchmarks, thermal and power soak tests, straggler and flapping-link detection — and automate them so that qualification is a repeatable pipeline, not a manual campaign.
- Drive cluster optimization end to end, tuning fabric configuration (adaptive routing, QoS and congestion control, SHARP in-network reduction, rail and topology-aware placement), collective communication libraries and algorithm selection, GPUDirect RDMA and storage paths, and host-level settings (huge pages, IRQ affinity, CPU governors, MIG and driver configuration) to convert raw hardware into delivered throughput.
- Partner with Infrastructure, Platform, SRE, and customer-facing teams to translate operational and customer performance needs into durable engineering solutions, and to feed diagnostic signal back into provisioning, remediation, and capacity workflows.