remotestar-team
Senior System Engineer (Munich, Germany)
Senior systems engineer based in Munich building the control plane for bare-metal AI/GPU infrastructure: Kubernetes operators and CRDs, GPU cluster scheduling, and driver/fabric tuning for distributed LLM training. Core stack: Python, Linux internals, Kubernetes, NVIDIA GPU tooling, Terraform/Ansible.
- Compensation
- Not disclosed
- Location
- Cambourne, United Kingdom of Great Britain and Northern Ireland
- Work setup
- Remote
- Seniority
- Senior
- Employment
- Contract
- Experience
- 10+ years
- Category
- Software engineering
- Posted
Skills
Must have
- 10+ years of software engineering experience with strong proficiency in Python; building system agents, APIs, and CLI tools
- Deep Kubernetes knowledge: CRDs, Operators, and Kubernetes API server architecture
- Hands-on experience managing NVIDIA GPU clusters; NVIDIA drivers, CUDA toolkit, NVIDIA Container Toolkit
- Deep understanding of Linux kernel, cgroups, namespaces, and system performance tuning
- Mastery of Terraform and Ansible with a focus on provisioning physical hardware
- Proven track record debugging complex distributed systems
- Legal authorization to work in the country where the position is based
Nice to have
- HPC background: Slurm, PBS, Volcano, Kueue, or Ray schedulers
- Bare metal provisioning: Cluster API (CAPI), Metal3, Tinkerbell, Canonical MaaS, or OpenStack Ironic
- High-speed networking: RDMA, InfiniBand, GPUDirect exposure to containerized workloads
- AI/ML familiarity: PyTorch Distributed, Megatron-LM, DeepSpeed, and LLM infrastructure requirements
- Observability: hardware health monitoring (DCGM) and distributed tracing for long-running jobs
Full description
About client :
Well-funded and fast-growing deep-tech company founded in 2019. We are the biggest Quantum Software company in the EU. They are also one of the 100 most promising companies in AI in the world (according to CB Insights, 2023) with 150+ employees and growing, fully multicultural and international.
Requirements
- Systems Programming Expertise: 10+ years of software engineering experience with strong proficiency in Python. You must be comfortable building system agents, APIs, and CLI tools.
- Deep Kubernetes Knowledge: You understand K8s internals beyond simple deployment. Experience with Custom Resource Definitions (CRDs), Operators, and the Kubernetes API server architecture.
- GPU Ecosystem Experience: Hands-on experience managing NVIDIA GPU clusters. Familiarity with NVIDIA drivers, CUDA toolkit, and the container runtime (NVIDIA Container Toolkit).
- Linux Internals: Deep understanding of the Linux kernel, cgroups, namespaces, and system performance tuning.
- Infrastructure as Code: Mastery of declarative infrastructure tools (Terraform, Ansible) but with a focus on provisioning physical hardware rather than just cloud VMs.
- Problem Solving: A proven track record of debugging complex distributed systems where the root cause could be code, network, or silicon.
Preferred qualifications
- HPC Background: Experience working with traditional supercomputing schedulers (Slurm, PBS) or modern batch schedulers (Volcano, Kueue, Ray).
- Bare Metal Provisioning: Experience with tools like Cluster API (CAPI), Metal3, Tinkerbell, Canonical MaaS, or OpenStack Ironic.
- High-Speed Networking: Knowledge of RDMA, InfiniBand, GPUDirect, and how to expose these technologies to containerized workloads.
- AI/ML Familiarity: Understanding of how distributed training works (e.g., PyTorch Distributed, Megatron-LM, DeepSpeed) and the infrastructure requirements of Large Language Models (LLMs).
- Observability: Experience building monitoring for hardware health (DCGM) and distributed tracing for long-running jobs.
Location: Applicants must have legal authorization to work in the country where the position is basedWhat you will be doing
- Building the Control Plane: Designing and developing the software layer (APIs, Controllers, Agents) that automates the lifecycle of bare-metal AI infrastructure.
- Orchestrating High-Scale Compute: Architecting scheduling solutions for large-scale distributed training jobs across massive clusters of GPUs (NVIDIA H200/B200/B300), ensuring efficient bin-packing and gang scheduling.