Integrate Amazon CloudWatch Metric Streams with Terraform

Our team, focused on re-engaging dormant users, requires a lot of experimentation to see what helps folks realize the value of our product. Because we need to spin up experiments and infrastructure and iterate quickly, we use Amazon Web Services (AWS) services including Amazon Lambda, Amazon Simple Email Service (SES), and a variety of others. Then, we rely on data from CloudWatch flowing into New Relic to help us monitor and gauge those AWS services.

We were using API polling for the CloudWatch data, but switching to CloudWatch Metric Streams and Terraform reduced cost per metric, saved time, and gave us access to complete CloudWatch metric data. Want to learn from our experiences? Here's how we did it.

Our challenges

We had set up API polling for the CloudWatch data, calling the server to check for changes at regular intervals, and then updating our client application. Polling for metrics like this was slow, costly, and we couldn't get at all the metrics-custom or otherwise-from CloudWatch.

We faced several challenges:

• 10-15 minute latencies for metrics
• Sampled and reduced dimension data
• Required custom New Relic namespaces for new metrics
• Cost of frequent calls to CloudWatch API

Our solution

Searching for a solution led us to Move Faster with New Relic One and Amazon CloudWatch Metric Streams, where we learned that we could switch from the API polling method to receiving streamed data from CloudWatch. Because our team also prefers an infrastructure-as-code approach, we wanted to find a way to have our solution in Terraform so we could make tracked changes to our infrastructure. Using Terraform, you can manage multiple entities in a command-line interface workflow.

We took a couple of shortcuts:

• Rather than rewriting the AWS CloudFormation template in Terraform, we opted to use Terraform to create and apply a CloudFormation stack. We put the change set from this CloudFormation template into a local variable in thefile.
• Then we created an resource and used the local variable for the template body. We did this because the stack template is unchanging, and we just wanted to pass parameters to it.

The results

Moving to CloudWatch Metric Streams gave us:

• Decreased latency
• Full CloudWatch metric data
• Mapping of CloudWatch metrics without custom namespaces
• Lower cost per metric

To try out this approach yourself, you can follow the steps in the next section.

How to set up CloudWatch Metric Streams with Terraform

Prerequisites:

• You must already have Terraform and AWS credentials set up.
• You need to have CloudWatch metric data that you want to stream to New Relic.

Public link

Please use the above public link if you want to share this noodl on another website.