Jobgether
Staff/ Principal Software Engineer (L6/L7): AI Platform Model Serving
- Compensation
- Not disclosed
- Location
- US
- Work setup
- Remote
- Seniority
- Principal
- Employment
- Full time
- Experience
- 12+ years
- Category
- Software engineering
- Posted
Skills
Must have
- Education and experience: Bachelor's, master's, or doctoral degree in Computer Science, Applied Mathematics, Engineering, or a related field, with at least 12 years of relevant experience in AI/ML inf
- Distributed systems expertise: Extensive experience designing and operating high-traffic distributed services, high-performance RPC systems, and production infrastructure with demanding availability,
- AI/ML model-serving experience: Demonstrated success building and scaling inference platforms for machine learning and foundation models that support predictive AI, generative AI, and other production
- Inference optimization: Strong understanding of latency and throughput optimization, resource allocation, GPU/CPU utilization, and techniques for improving inference efficiency and reducing infrastruc
- Modern inference technologies: Practical experience with model-serving platforms and runtimes such as NVIDIA Triton Inference Server, vLLM, SGLang, TensorRT-LLM, Text Generation Inference (TGI), DeepS
- Cloud-native infrastructure: Experience deploying and scaling AI workloads using Kubernetes and public cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Distributed serving and control planes: Familiarity with model gateways, intelligent routing, control-plane architecture, experimentation infrastructure, and A/B testing for production AI systems.
- Technical leadership: Ability to set multi-year technical roadmaps, lead complex architectural initiatives, and influence decisions across teams without relying on formal authority.
- Communication and collaboration: Strong written and verbal communication skills, including the ability to explain complex technical concepts to non-technical stakeholders and produce clear design prop
- Business and product alignment: Proven ability to work with product managers, operations teams, and senior leadership to translate business objectives into scalable, maintainable technical solutions.
Nice to have
- Contributions to open-source projects related to AI/ML, distributed systems, model serving, or infrastructure.
- Hands-on experience developing AI models or implementing optimization techniques such as quantization, model compression, and model pruning.
- Proficiency in using AI-assisted coding tools to improve software development productivity.
- Experience with intelligent model gateways that select models or providers based on request context, performance requirements, or cost considerations.
- Exposure to multi-adapter model serving, disaggregated prefill and decode, and large-scale key-value (KV) cache management.
- Experience with evaluation frameworks and benchmarking tools for large language models and AI agents.
- Knowledge of GPU topology, NVIDIA CUDA programming, NCCL, machine learning compilers, and performance profiling tools such as NVIDIA Nsight.
- Familiarity with deep learning frameworks and ecosystems, including PyTorch, TensorFlow, JAX, and Hugging Face Transformers.
Full description
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Staff/Principal Software Engineer (L6/L7): AI Platform Model Serving based in the United States, with remote work flexibility.
In this highly strategic engineering role, you will shape the next generation of AI and machine learning model-serving infrastructure at global scale. You will lead the architecture and development of a unified platform supporting predictive AI, generative AI, foundation models, and agentic workloads across real-time and asynchronous environments. Working at the intersection of distributed systems, GPU optimization, and advanced inference technologies, you will tackle complex challenges in performance, reliability, scalability, and cost efficiency. You will help establish technical standards, influence long-term engineering strategy, and guide multiple teams through architectural decisions. The role offers substantial research and development opportunities in emerging AI infrastructure, including intelligent model routing, speculative decoding, and multi-GPU inference. You will collaborate with senior technical leaders and cross-functional partners to build durable, high-performance systems that enable innovative AI-powered experiences.
Accountabilities
- Lead platform architecture and technical strategy: Define and drive the architecture of a unified AI model-serving platform that integrates internal inference infrastructure, third-party model hosting, and external large language model providers through consistent serving, routing, and experimentation capabilities.
- Build high-performance inference infrastructure: Design and optimize distributed systems capable of supporting large-scale production workloads, from low-latency predictive models to foundation models and complex generative AI applications.
- Optimize GPU utilization and inference economics: Develop and implement advanced techniques such as disaggregated serving, speculative decoding, prompt caching, intelligent routing, multi-GPU inference, and efficient capacity allocation to reduce latency and cost per inference.
- Develop intelligent model-serving capabilities: Enable teams to deploy, route, evaluate, and A/B test models consistently across different runtimes, infrastructure environments, and providers.
- Advance asynchronous and agentic workloads: Design reliable, resumable execution systems for long-running AI agents and asynchronous inference tasks, ensuring fault tolerance, operational consistency, and scalability.
- Drive technical innovation and research: Evaluate emerging AI serving technologies, identify infrastructure bottlenecks, and translate research and experimentation into production-ready solutions.