Senior Software Engineer ML API AI Inference

You will design, implement, test, and operate ML API features for a disaggregated AI inference platform. You will integrate models and serving components, improve routing, performance, reliability, and release quality, and lead technical delivery across multiple components. You will also mentor engineers and improve developer and customer workflows.

Responsibilities

• Design, implement, test, and operate features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, and multimodal inputs
• Write design documents, evaluate tradeoffs, plan milestones, and coordinate multi-component delivery
• Implement consistent request and response behavior across inference backends
• Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features
• Extend inference services with vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components
• Implement routing, state transfer, error handling, retries, cancellation, and request lifecycle management
• Diagnose bottlenecks and improve latency, throughput, and resource efficiency
• Build conformance, validation, integration, and performance-regression tests
• Improve metrics, tracing, logging, alerts, diagnostic tools, resilience, and incident response
• Improve configuration, documentation, examples, and debugging workflows
• Review designs and code, mentor engineers, and remove delivery roadblocks

Requirements

• 5+ years of software engineering experience
• Strong Python programming ability
• Proficiency in Go, C++, Rust, or another production-services language
• Experience building production APIs
• Knowledge of REST, gRPC, or streaming protocols
• Experience with Linux, containers, Kubernetes or comparable orchestration, and CI/CD
• Experience diagnosing distributed-system correctness, reliability, and performance issues
• Bachelor's degree in a related discipline or equivalent practical experience