Principal Systems Engineer: Linux Kernel
Own test and validation infrastructure for Cedana's kernel-level GPU checkpointing system that transparently migrates AI/HPC workloads (works with Kubernetes, SLURM, NVIDIA Dynamo). Deep Linux kernel work: root-cause defects at the kernel/driver boundary, optimize performance with perf/eBPF, and build automated performance and regression harnesses.
- Compensation
- $140K–$180K / yr
- Location
- San Francisco
- Work setup
- Not specified
- Seniority
- Principal
- Employment
- Not specified
- Experience
- 5+ years
- Category
- Software engineering
- Posted
Skills
Must have
- 5-10 years of software engineering experience with Linux Kernel
- Kernel-level depth: reads kernel and driver source, root-causes defects at the kernel/driver/syscall boundary
- Expert-level mastery of at least one performance- or correctness-critical Linux systems domain (networking data plane, virtualization/hypervisor, storage and block I/O, scheduling, memory management,
- Performance and latency engineering: characterizes throughput, jitter, tail latency; uses tracing/profiling tooling (perf, ftrace, eBPF) to localize bottlenecks
- Systems-level QE: designs and owns automated test infrastructure, performance and regression harnesses; reproduces kernel-level races and corner cases
- Enterprise Linux distribution environment (RHEL or equivalent): version matrices, backports, customer-grade triage
- Principal scope: owns test or validation strategy for a subsystem; sets technical direction; mentors
- Excellent communication: writes concise, articulate design papers; responsive to internal SLAs
Nice to have
- Experience with OVS, DPDK, SR-IOV, RDMA, or high-performance packet processing
- Experience with CRIU + QEMU live migration + VFIO
- Experience with runc / containerd / OCI / namespaces / cgroups v2 / overlayfs
- Real-time Linux (PREEMPT_RT) and deterministic-path validation
- Device virtualization beyond NICs: VFIO, vGPU / GPU passthrough
- Virtualization internals: KVM / QEMU / libvirt
- Container and Kubernetes networking (CNI, OVN-Kubernetes)
- Upstream kernel or open-source contribution history
- Telco / 5G / NFV domain context
- Credentials as proxies, not gates: RHCA / RHCE, ISTQB CTAL-TM
Full description
Compensation: $140k – $180k
Introducing CedanaThe Problem
AI and HPC infrastructure suffer from scarcity and high costs, so when failures occur, they are costly in time and money. Cluster productivity directly determines research output and revenue. Achieving high utilization and throughput is increasingly challenging due to the complexity of workloads, hardware, and operations.
Cedana’s Solution
Cedana maximizes AI+HPC cluster utilization and reliability with automated GPU checkpointing infrastructure. We enable transparent, fast migration of GPU workloads across instances without losing work. Workloads automatically migrate to achieve new levels of reliability and throughput while accelerating time to results. Our system is at the kernel/OS level, requiring no code or config changes, and works seamlessly with Kubernetes, SLURM, and NVIDIA Dynamo. Today, we're deploying into leading inference platforms, neoclouds, enterprise, and research clusters.
The Team
Cedana's founding team has spent over a decade making computation run fast, productively, and reliably for AI. Our research appears in NeurIPS and CVPR. We published some of the earliest formal methods for guaranteeing convergence in distributed training. At Shopify, we've developed a control plane for robotics fleets used in warehouse automation. We bring repeat founder experience, having built and exited a Series B healthcare AI company.
Backed by Y Combinator, Initialized Capital, Pebblebed (founders of OpenAI and Facebook AI Research), Keith Adams (engineer #20 at VMware, founded HHVM and FAIR at Facebook, Chief Architect at Slack), Venture Guides, Garry Tan, and Gokul Rajaram.
The RoleWhat you’ll own
You will own the core components of our test and validation for kernel development. This includes measurement and optimization. Over time, you will contribute to critical core components of our system, with a focus on new capabilities, performance, and reliability.
What You'll Do
- Validate and test automation: Our engineers contribute to and develop testing capabilities. This will be a core part of your initial work to establish your understanding of how our system works. Reliability and testing is a key part of our culture.
- Measure and optimize platform performance: Get Cedana to the theoretical maximum performance by understanding fundamental bottlenecks. Measure reliability, throughput and performance using our internal tools.
- Design and contribute to key system components : Our solution touches all the major aspects of the OS, kernel, GPU, and CPU. Write design papers to outline your vision for a specific capability and then lead implementation.
- Educate our team: Our team is our best learning resource and we continually educate each other. We huddle and co-pair as needed.