01/19/2022 | News release | Distributed by Public on 01/19/2022 11:05
At New Relic, we believe that programmatically tracked service level indicators (SLIs) are foundational to our site reliability engineering practice. When we have programmatic SLIs in place, we lessen the need to manually track performance and incident data. We're able to reduce that manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically-hence "programmatic."
Programmatic SLIs have three key characteristics:
In this post, we'll explain how New Relic site reliability engineers (SREs) help their teams develop and create programmatic SLIs.
SLIs-identifying capabilities
An important part of creating programmatic SLIs is identifying the capability of the system or service for which you're creating the SLI. At New Relic, we use the following definitions:
You can create SLIs at any layer, but this post will focus primarily on system-level SLIs.
SLIs and SLOs-indicators and objectives
But first, we need some more definitions. An indicator is something you can measure about a system that acts as a proxy for the customer experience. An objective is a goal for a specific indicator that you're committed to achieving.
Indicator | Objective |
X should be true... | Y portion of the time. |
Configuring indicators and objectives is the easy part. The hard part is thinking through what measurable system behavior serves as a proxy for customer experience. When setting system-level SLIs, think about the key performance indicators (KPIs) for those systems, for example:
Your indicators and objectives should provide an accurate snapshot of the impact of your system on your customers.
A more precise description of the indicator and objective relationship is to say that SLIs are expressed in relation to service level objectives (SLOs). When you think about the availability of a system, for example, SLIs are the key measurements of the availability of the system while SLOs are the goals you set for how much availability you expect out of that system. And service level agreements (SLAs) explain the results of breaking the SLO commitments.
SLI | SLO | SLA |
X should be true... | Y portion of the time, | or else. |
So, as an example, consider the routing capability of a data-ingest tier. A plain-language definition for the data routing capability might look like: "Incoming messages are available for other systems to consume off the message bus without delay." With that definition then, we might establish the SLI and SLO as, "Incoming messages are available for other systems to consume off of our message bus within 500 milliseconds 99.xx% of the time."
Create programmatic SLIs
You should write your programmatic SLIs in collaboration with your product managers, engineering managers, and individual contributors who work on a system. To define your programmatic SLIs (and SLOs), apply these steps:
Example capabilities and definitions
Here are two example capabilities and definitions for an imaginary team that manages an imaginary dashboard service:
Capability: Dashboards overview.
Availability Definition:Customers are able to select the dashboard launcher, and see a list of all dashboards available to them.
Capability: Dashboards detail view.
Availability Definition: Customers can view a dashboard, and widgets render accurately and timely manner.
To express these availability definitions as programmatic SLIs (with SLOs to measure them), you'd state these service capabilities as:
Automatically set up SLIs and SLOs with New Relic One
Managing all of these service level definitions and visualizations can be difficult if you need to start from scratch. It's like writing a long paper-it's always harder to get started when the page is blank.
Luckily, New Relic One's service level management functionality can help identify SLIs to start measuring and allow you to establish a baseline for SLOs. For example, the tool identifies the most common SLIs for a given service, most often some measurement of availability and latency, and it scans the historical data from a service to determine the best initial setup. Across the platform, you'll find ways to automatically set up SLIs, like we show here, or you can manually create them with NRQL queries.
If you're looking for a one-click setup to establish a baseline for SLIs and SLOs in New Relic, just follow these steps: