Staff Software Engineer

• We are looking for a Staff Software Engineer - SRE to help us support our highest value Grafana Cloud customers by increasing the reliability of our Cloud databases that are based on Mimir, Loki, Tempo, and Pyroscope. We provide these databases as a SaaS product from AWS, GCP, and Azure across all regions
• The SRE team is embedded within the Mimir, Loki, and Tempo squads and focuses on ensuring that Grafana Cloud’s database products deliver exceptional reliability for our highest-SLA customers. In this role, you will:
• Partner closely with product engineering squads (embedded model)
• Own production reliability for high-SLA and complex customer environments
• Design and implement automation to scale our reliability practices
• Ensuring our customers meet our SLO targets
• Define and evolve per-tenant SLOs and reliability models
• Proactively reduce SLO burn to prevent repeat incidents
• Serving as a primary escalation point and on-call for relevant incidents
• Lead customer-impacting incident response and post-incident reviews
• Contribute to design docs and code reviews
• Influence feature design to ensure production scalability and operability
• Build automation to eliminate toil where needed
• Improve alert quality and reduce noisy escalations
• We seek a staff software engineer operating at the intersection of customer needs, production systems, and product engineering
• Regular 1:1s to with your manager and colleagues
• Reviewing and creating SLOs, proactively investigating ways in which we can further reduce budget burn for those SLOs, which can be self-directed or as the result of learnings from incidents, and may include improvements to monitoring, automation, increasing self-healing, auto-scaling, etc
• Improve observability of customers within their environments
• Designing and implementing solutions to ensure reliability and scalability of our environments can meet rapidly increasing demands
• Develop fault-tolerant design patterns ensuring that we are considering reliability at all stages of the service lifecycle
• Collaborating with our Engineering Leaders to help define and influence product strategy, roadmaps and technical designs
• Participate in PR review and collaborating with other engineers on their Design Docs
• Teach others about Site Reliability Engineering and communicate best practices to be applied early in development of new features and functionality
• Participate in Incident Response when applicable, including investigation through to resolution, PIR, and communication with customers via Bridge calls where necessary

Benefits

• Vacation: Balance is key. Our team enjoys 30 days of paid vacation each year on top of national holidays, parental leave, and sick leave. We also take a breather on a number of Grafana Shutdown Days each year