Site Reliability Engineer (High Performance Computing)
Runs SpaceX's shared high-performance computing platform as a product: keeping Linux clusters, storage, and user-facing services reliable through infrastructure as code, observability, automation, and a sustainable incident process. Core stack: Linux, IaC (Ansible/Terraform), Prometheus/Grafana, Python, containers/Kubernetes, HPC schedulers.
- Compensation
- $125K–$195K / yr
- Location
- Hawthorne, CA
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Sre
- Posted
Skills
Must have
- Bachelor's degree in computer science, engineering, math, or a scientific discipline; OR 2+ years of professional experience operating production infrastructure
- 2+ years of experience with Linux operating systems in production
- 2+ years of experience operating production infrastructure (servers, services, or networks), including monitoring, debugging, and repairing what you own
- Participate in on-call rotation and work extended hours/weekends as needed
- Based in Hawthorne, CA, primarily on-site
- Must be a U.S. citizen/national, lawful permanent resident, refugee, or asylee (ITAR)
Nice to have
- Eligibility for access to classified material up to TS/SCI with polygraph
- 2+ years of professional experience in SRE, DevOps, or production infrastructure engineering
- Experience with monitoring and alerting (Prometheus, Grafana, Nagios, or similar)
- Experience deploying and maintaining configuration management or infrastructure as code (Ansible, Puppet, Terraform, or similar)
- Experience writing scripts/code (e.g. Python) to automate common tasks
- Experience with containers (Docker, Podman, Singularity/Apptainer)
- Experience with Kubernetes administration for on-premise deployment
- Experience with distributed or high-performance storage (VAST or similar), including capacity, performance, and lifecycle management
- Familiarity with HPC clusters, schedulers (Slurm, PBS, LSF), or GPU compute
- Familiarity with scientific computing (CFD, FEA) and/or ML training workloads (PyTorch, TensorFlow, CUDA) and/or AI inference workloads
- Good understanding of version control, testing, continuous integration, build, deployment and monitoring
- Ability to communicate clearly with users, peers, and vendors in incident and design settings
- Comfortable working with mission-critical and sensitive systems
Full description
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today SpaceX is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars.
SITE RELIABILITY ENGINEER (HIGH PERFORMANCE COMPUTING)
SpaceX HPC is a shared compute platform used across the company — vehicle and structures simulation, machine learning, AI inference, and more. We support every program at SpaceX to design and operate the worlds most advanced rockets and satellites. This role exists to put a real Site Reliability Engineer operating model on these capabilities and accelerating the world class engineering at SpaceX: toil reduction, automation, observability, and a sustainable incident process.
We are looking for a Site Reliability Engineer who wants to own everything from Linux machines and our Infrastructure as Code, storage, and user facing applications – the whole ecosystem as a product, not as a ticket queue. You do not need a prior HPC title. You do need production instincts — you have operated real infrastructure, you write code to delete toil, and you care about whether users can actually get work done, not just whether nodes ping. You’ll work alongside HPC systems engineers who design and commission clusters to help make them more reliable and provide world class services for world class engineers.
Aerospace experience is not required. We value engineers who treat teammates with fairness and respect, who are self-critical, and who will take ownership of hard production problems.
RESPONSIBILITIES:
- Participate in the team's on-call rotation; practice sustainable incident response and blameless postmortems
- Manage node lifecycle with infrastructure as code: OS images, firmware, configuration management, kernel and driver stack
- Build observability for both HPC administrators and end users — cluster, node, and storage health for operators, and job/workflow-level signal for the people running work on the platform
- Reduce toil with automation; split time between operating production systems and writing the software that makes that work smaller
- Sustainably manage resources, including compute and storage
- Lead capacity planning with users across the company: understand what they will need next, and turn that into a concrete picture of tomorrow's compute and storage
- Collaborate with HPC systems engineers and with engineers across all disciplines across the company on operable, maintainable infrastructure
BASIC QUALIFICATIONS:
- Bachelor's degree in computer science, engineering, math, or a scientific discipline; OR 2+ years of professional experience operating production infrastructure in lieu of a degree