Senior Software Engineer - AI Infrastructure

Overview

Join our team within AI Infrastructure, where we are building the AI data-plane that powers all LLM inferencing workloads across Microsoft and Azure customers—from cutting-edge startups to Fortune 500 enterprises. Our converged AI fabric delivers inference capabilities for all LLMs in Microsoft catalog, including OpenAI, Anthropic, Mistral, Cohere, Llama, and more.

As a Senior Software Engineer, you will shape the future of one of the largest and fastest-growing services in Azure, foundational to Microsoft’s AI strategy. Our mission is to serve models at scale—reliably, efficiently, and with ultra-low latency—enabling a rich set of AI-powered product experiences. This is a rapidly evolving space with immense opportunities to learn, innovate, and drive industry-wide impact!

Responsibilities

• Design and implement highly scalable, reliable, and performant distributed systems that power Azure AI traffic management and scheduling.

• Drive architecture and technical direction for routing, capacity allocation, and workload optimization services.

• Develop algorithms and control-plane systems that intelligently distribute AI workloads across global infrastructure while meeting latency, availability, and cost objectives.

• Collaborate with partner teams to onboard new models, regions, and customer offerings while maintaining operational excellence.

• Build observability and operational tooling to monitor service health, capacity utilization, traffic patterns, and customer impact in near real-time.

• Lead investigations into complex production issues and drive systematic improvements to reliability, resiliency, and platform automation.

• Identify and execute opportunities to improve fleet efficiency, reduce operational overhead, and optimize utilization of premium AI infrastructure.

• Mentor engineers, review designs and code, and raise engineering standards across the organization.

• Drive engineering excellence through testing, automation, documentation, and operational best practices.

• Influence long-term platform strategy and contribute to roadmap planning for next-generation AI infrastructure.

QualificationsRequired Qualifications:

• Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

Preferred Qualifications:

• Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.