Senior Site Reliability Engineer - Storage
Build and maintain Lambda’s high-performance storage infrastructure, ensuring reliability and scalability for AI workloads while automating incident response and fleet management.
- Compensation
- $267K–$356K / yr
- Location
- San Francisco Office (Fremont St)
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 5+ years
- Category
- Sre
- Posted
Skills
Nice to have
- Experience with cutting-edge software-defined storage solutions such as VAST or Weka.
- Enterprise storage expertise in solutions such as NetApp, Dell PowerScale, GPFS, or Lustre.
- Experience writing or operating Kubernetes CSI drivers.
- Experience with SR-IOV and virtualization (KVM/QEMU).
- Experience with GPUDirect Storage, RDMA, InfiniBand, or RoCE networking.
- Familiarity with NIC-level diagnostics (ethtool, mlxlink) and fleet-wide operations tooling (clush or similar).
- Contributions to open-source storage projects.
Full description
Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU.
If you'd like to build the world's best AI cloud, join us.
*Note: This position requires presence in our San Francisco or San Jose office location 4 days per week; Lambda’s designated work from home day is currently Tuesday.
Lambda's Storage Engineering team is the backbone behind our world-class storage offerings, operating at a scale seldom seen in today's markets. We own the full spectrum of Lambda's data platform services—from low-level storage systems to the APIs and tooling our customers build on every day.
Our work directly powers some of the most demanding compute workloads in the industry, which means reliability and performance aren't just goals—they're the baseline. We're looking for engineers who want to solve problems at scale, own critical systems end to end, and help shape the future of infrastructure for AI and beyond
What You’ll Do
- Own the reliability, performance, and capacity health of Lambda's production storage fleet across all data centers, operating behind Lambda's own software-defined data plane.
- Build and maintain monitoring, dashboards, and alerting for storage performance, capacity, and hardware failures.
- Investigate and resolve storage-related incidents using deep telemetry, logs, and performance profiling — from a single flapping NIC to a cluster-wide rebuild.
- Automate ticketing, escalation, and incident-response workflows so the team spends less time on repetitive triage and more time on root cause.
- Design and maintain self-healing automation for common failure modes: drive replacement, node swaps, rebuild monitoring, and capacity rebalancing.
- Implement CI/CD pipelines for storage automation and tooling.
- Partner with Storage Engineers, Fleet Orchestration, and Release Engineering to automate the deployment and configuration of software-defined storage across new and existing sites using tools such as Ansible, Jenkins etc.
- Work with hardware and networking teams to diagnose low-level I/O and network issues — NIC errors, RDMA/RoCE/InfiniBand fabric health, path multipathing — that surface as storage-layer symptoms.
- Participate in an on-call rotation supporting Lambda's storage fleet, with a focus on driving down MTTR and building the automation that keeps you from getting paged for the same thing twice.