Senior Software Engineer, AI Infrastructure
Designs, builds, and operates services that route and prioritize AI requests (e.g., Azure OpenAI, Copilot) across Microsoft's global GPU infrastructure. Day-to-day involves distributed systems design, coding in C#/Java/Go, production on-call, and scaling cloud-based AI traffic management.
- Compensation
- $119.8K–$234.7K / yr
- Location
- United States, Washington, Redmond
- Work setup
- Onsite
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 4+ years
- Category
- Software engineering
- Posted
Skills
Must have
- Bachelor's Degree in Computer Science or related field AND 4+ years engineering experience building distributed systems with coding in C#, Java, or Go (or equivalent experience)
- Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check
Nice to have
- Master's Degree in Computer Science or related field AND 6+ years experience building distributed systems with C#, Java, or Go (or Bachelor's AND 8+ years, or equivalent experience)
- 3+ years running distributed services on public cloud platforms such as Azure, AWS, or GCP
- 2+ years experience with systems that route/schedule traffic or allocate computing resources
- Experience with Python, Docker, and Kubernetes
- Experience with AI/ML infrastructure, such as serving models in production or managing GPU fleets
- Experience delivering projects with partner teams from design through launch and operation
- Experience being on call for highly available services and improving monitoring to reduce incidents
Full description
Overview
Every time someone uses Azure OpenAI Service, their request needs to reach the right graphics processing unit (GPU), in the right region, at the right moment. Our team makes that happen. We build the platform that decides how AI traffic flows across Microsoft's global infrastructure, so services like Azure OpenAI and Microsoft Copilot stay fast and reliable. You'll join a supportive, connected team that keeps these experiences fair and dependable for every customer while making the most of every GPU we run.
As a Senior Software Engineer, AI Infrastructure on our team, you'll design, build, and run services that route and prioritize AI requests and match them to available computing capacity. You'll own the design of key parts of our platform, work with partner teams to bring new AI models and capabilities to customers, and help keep our services healthy in production. This opportunity will allow you to deepen your knowledge of distributed systems at global scale, learn how the world's leading AI models are served, and grow as a technical leader and mentor on your team.
Microsoft's mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
- Design - Own the design of key parts of our platform for routing AI traffic and managing computing capacity. Write design proposals, compare options, and make sure designs meet goals for speed, reliability, scale, and cost.
- Coding - Write clear, secure, well-tested code and set a good example for the team. Give thoughtful, timely code reviews that help others grow.
- Implement - Break work into clear, dependable milestones with program managers. Use feature flags and gradual rollouts to test changes safely, measure their impact, and roll back quickly when needed.
- Reliability and Supportability - Take part in the on-call rotation as a designated responsible individual (DRI). Add monitoring and dashboards, improve troubleshooting guides, and help reduce how often incidents happen.
- Understand User Requirements - Work with product managers and partner teams to understand what customers need and turn those needs into clear technical requirements.
- Engineering Excellence - Build security, privacy, and compliance into every stage of your work, from design to deployment. Automate testing and releases, and share what you learn with the team.
- AI-Native Development - Use artificial intelligence (AI) tools across the software development lifecycle (SDLC), from writing code and tests to investigating live issues. Take responsibility for AI-generated work and help teammates do the same.