PagerDuty Inc.

05/02/2024 | Press release | Distributed by Public on 05/02/2024 08:54

IRL to IAC: Your Environment to PagerDuty via Terraform

IRL to IAC: Your Environment to PagerDuty via Terraform

by Mandi Walls
May 2, 2024

Introduction

Figuring out how to represent your as-built environment in PagerDuty can be confusing for new users. There are a lot of components to PagerDuty that will help your team be successful managing incidents, integrating with other systems in your environment, running workflows, and using automation. Your organization might have a lot of these components - users, teams, services, integrations, orchestrations, etc.

To manage larger or more complex PagerDuty environments in a programmatic fashion, we recommend using Infrastructure as Codetools like Terraformor Pulumi. Both solutions have PagerDuty providers available. In this example, we'll look at Terraform and you can find their PagerDuty provider here. (We'll look at Pulumi in a future post.)If you're new to Terraform, HashiCorp has documentationto help you get started and there are lots of great intro videos on YouTube, including this tutorial by TechWorld with Nana.

Getting started with PagerDuty is a prescriptive process. Certain structures have to be created in a particular order when creating a new account, and there will be steps in the web UI to guide you. For the examples below, I'm assuming you have a PagerDuty account and your users are already onboarded. For more information on getting started with PagerDuty, be sure to check out our online learning at PagerDuty University.

Goals

For this foray into Terraform, I'm going to assume that a production environment is already running. I'm going to use the services included in GCP's microservices-demoproject to give me something to work with. There is enough complexity in this demo to create services, dependencies, and integrations! The service diagram in the README will be helpful for defining services and their relationships.

When I'm done, I want to have a full set of services mimicking the Microservices Demo environment, with defined dependencies. Each service will also have a generic integration endpoint defined so they can receive alerts.

You can view all of the code for this example in my GitHub account.

Basic info you'll need

I'm going to include my team members as local data in my Terraform files. If your team is already using Terraform, you might be using data modules or local files to hold data, and that's fine too. If you are planning to use Terraform to manage users, check out the provider docs for pagerduty_user.

To include your users as local data, you'll only need their user IDs.

Ordering of resources

Once a PagerDuty account has users, the next steps are to create at least one schedule and escalation policy. We do this before creating any services, since having an escalation policy is a requirement for building a service. You can build a new escalation policy for each service, but I'm going to create my escalation policies explicitly. Each escalation policy has at least one schedule included. You can also create escalation policies with individuals.

Schedules and Escalation Policies

Here's what one of my schedules looks like, including the local data for the team members, using the pagerduty_scheduleresource: 

locals{ dbre_team =["PC6K5C9","PULO4NW"] demo_team ="POJW28N" app_team  =["PC6K5C9","PULO4NW","P73R26T"]}resource"pagerduty_schedule""msd_apps_sched"{ name ="ApplicationTeamMicroservicesDemo" time_zone ="America/New_York" teams =[local.demo_team] layer{ name ="ApplicationDevelopers" start ="2024-01-01T00:08:00-05:00" rotation_virtual_start ="2024-01-01T00:08:00-05:00" rotation_turn_length_seconds =604800 users =local.app_team }}

Only a few arguments are required: the time_zone(see the PagerDuty API docsfor supported zones); and at least one layerthat requires start, rotation_virtual_start, rotation_turn_length_seconds, and users.

You can also include restrictions and other features in your schedules. See the provider docs for more information.

I can then reference this pagerduty_scheduleresource when I create my escalation policy, using the pagerduty_escalation_policyresource: 

resource"pagerduty_escalation_policy""msd_apps"{ name ="ApplicationTeamMicroservicesDemo" num_loops=1 teams  =[local.demo_team] rule{ escalation_delay_in_minutes=10 target{ type="schedule_reference" id =pagerduty_schedule.msd_apps_sched.id } }}

The pagerduty_escalation_policyresource also has only a few requirements: a nameand one or more ruleblocks that contain escalation_delay_in_minutesand a target. The targetrequires an id, which is where I've linked it to the schedule I've defined above: pagerduty_schedule.msd_apps_sched.id.

Services

Now that I have an escalation policy, I can create my services. Looking at the documentation for the microservices demo, there are 12 potential services in the environment. I'm going to leave off the loadgeneratorservice, so I'll create 11 technical services to match the environment. I'm also going to create one business serviceto represent the entire demo environment in my service graph.

I'm using the most basic configuration for my services. You can check out the provider docsfor all of the resource arguments. I can always modify my services later with Terraform if I need to change or add something.

Services really only require a name, and an escalation_policy. The alert_creationargument is deprecated at the time of this writing with the new setting being create_alerts_and_incidentsfor all services, so you may not need it in your code. I'm going to create all of my services the same to start, so they will all look like this: 

#RedisCacheresource"pagerduty_service""Demo_Redis_Cache"{ name ="RedisCache-MicroservicesDemo" escalation_policy=pagerduty_escalation_policy.msd_dbre.id alert_creation ="create_alerts_and_incidents"}#ShoppingCartresource"pagerduty_service""Demo_Cart"{ name ="ShoppingCart-MicroservicesDemo" escalation_policy=pagerduty_escalation_policy.msd_apps.id alert_creation ="create_alerts_and_incidents"}

Because my escalation_policyresources were created with Terraform, I can include their idsdirectly via the resource reference. I can use different escalation policies for each service if I want to as well, but here I've only created two - one for the DBRE team and one for the Application team.

As I mentioned above, I'm also going to create a business serviceto represent the application stack as a whole. Business services help other parts of my organization identify when incidents occur that impact user-facing applications, and they will appear on the Service Graph page. You can think of the business service as representing the Useron the original environment diagram!

The pagerduty_business_serviceresource looks a bit like the resource for technical services, but it does not have an associated escalation_policy: 

resource"pagerduty_business_service""Microservices_Demo"{ name ="MicroservicesDemo" description="ServicesalignedbehindtheOnlineBoutiqueDemo" team =local.demo_team}

There are only a couple of additional arguments available for the pagerduty_business_serviceresource, you can see them in the provider docs.

Service Dependencies

Once I have all the services for my environment defined, I can add their relationships using service dependencies. Dependencies will help my team determine the potential impact of incidents across services.

Each dependency has to be defined individually as a relationship between a dependent_serviceand a supporting_service. Using the convention in the graph diagram in the Microservices Demo README, a dependent_servicewill be above all of its supporting_services, but below all of the services that depend on it. I will also create a dependency between the Frontendservice and my Microservices Demobusiness service.

The following resource creates the dependency between my Frontendand Checkoutservices: 

resource"pagerduty_service_dependency""fe_to_checkout"{ dependency{ dependent_service{ id =pagerduty_service.Demo_Frontend.id type=pagerduty_service.Demo_Frontend.type }
 supporting_service{ id =pagerduty_service.Demo_Checkout.id type=pagerduty_service.Demo_Checkout.type } }}

I can reference the idand typecharacteristics of these services since they were both created with Terraform.

The dependency that links the business service to my technical services looks slightly different because of the pagerduty_business_serviceresource: 

resource"pagerduty_service_dependency""biz_to_fe"{ dependency{ dependent_service{ id =pagerduty_business_service.Microservices_Demo.id type=pagerduty_business_service.Microservices_Demo.type } supporting_service{ id =pagerduty_service.Demo_Frontend.id type=pagerduty_service.Demo_Frontend.type } }}

When all of my service dependencies are defined, I can use the web UI to compare them with the original application topology diagram from the README. I can move the objects around on the graph to line up like the original:

Integrations

Finally, I want to enable all of my services to receive alerts. There are a lot of different types of integrationsthat can be configured on PagerDuty services, but I'm going to start with a basic Events API v2integration for each service. You can find more information about how to define other types of integrations, including email integrations and referencing specific vendors, in the provider docs. 

resource"pagerduty_service_integration""events_msd_frontend"{ name ="APIV2" type ="events_api_v2_inbound_integration" service =pagerduty_service.Demo_Frontend.id}

Next Steps

Now that my services are defined in PagerDuty and they all have integrations to receive events, I'm ready to roll with basic functionality. I can also use these basic resources to incorporate helpful solutions like event orchestrationsand automation actionsas well as more sophisticated integrations to meet the needs of my team.

For the latest information on the PagerDuty Terraform provider, always check the provider documentation. Our maintainer, José Antonio Reyes, presents Terraform Time on the PagerDuty Twitchon Wednesdays at 4pm Eastern, and hosts a quarterly roundtable. If you have other questions, we'd love to help! You can reach us at [email protected].