What Exposed OPA Servers Can Tell You About Your Applications

This blog entry discusses what we've discovered after identifying hundreds of exposed OPA servers via Shodan and how exposed OPAs can negatively impact your applications' overall security.

The need for policy-as-code

In today's agile world that includes sprints, scrum, containers, DevOps and DevSecOps, and the cloud, it is hard for network, security, and compliance teams to keep up with development teams and business demands without the use of automation. That's why approaches such as infrastructure-as-code (IaC), GitOps, and security-as-code (SaC) have become essential in protecting cloud environments and microservices. All these approaches work together to achieve cloud security - helping integrate security at the beginning of processes and deployments to keep threats and risks at bay.

But what about compliance? How can you check, enforce, and continuously monitor your systems to ensure that they are following security best practices and are complying with the latest security standards, such as Payment Card Industry Data Security Standard (PCI-DSS), Health Insurance Portability and Accountability Act (HIPAA), and System and Organization Controls (SOC 2)? This is where policy-as-code comes into play and can significantly help speed up the compliance creation, automation, and verification processes.

Policy-as-code (or compliance-as-code) has become a great way to implement, manage, and track changes to systems' policies and ensure that they are applied and enforced across the 4C's of cloud-native systems: code, containers, clusters, and cloud.

Using a similar approach to IaC and SaC, policy-as-code uses all the great tools and techniques already used in software development to address compliance concerns. For one, policies are created and stored in specific file formats such as JSON or YAML or using a particular programming language such as Python or Rego. These are stored in a Git repository for version control, tracking, and auditability. By doing this, organizations can develop and trigger pipelines and automation on these repositories whenever there are changes. Policy-as-code can be programmatically implemented in code reviews, automated testing, and continuous integration and delivery (CI/CD) pipelines, ensuring that changes are appropriately validated and tested before reaching production systems. OPA is one of the leading open-source tools available for the policy-as-code process. It can be used for many different applications and systems on the web. Still, one of its leading adopters has been an Admission Controller for Kubernetes Clusters, where its Gatekeeper version works as a bouncer for any containers trying to run inside the cluster.

Exposed servers on Shodan

During our analysis of exposed cloud-native tools, specifically, publicly exposed kubelets, we have also identified almost four hundred exposed OPA servers running on port 8181 via Shodan, and this number has been increasing over the past few months. Port 8181 is the default port for OPA servers, and they were all with the API endpoint available for unauthenticated and unauthorized requests. Based on our Shodan search, the United States, the Republic of Korea, and Germany are the top three countries with the most significant number of exposed OPAs.

Figure 2. Exposed OPA servers found on Shodan as of August 11th, 2022. Query: port:8181 "policy-enable"

We used the list policies endpoint to gather information about all the policy modules installed in that instance, as indicated in the official OPA documentation. We then queried each 389 open OPA servers to identify, collect, and analyze what kind of sensitive information they could expose via their policy module information. We primarily focused on the Policy API to list policies and analyze data.

First, we analyzed the version of the OPA installed on these servers by querying the /v1/config/ endpoint. As seen on Figure 3, most of the exposed servers were using outdated versions of OPA, such as 0.34.2. As of writing, the latest OPA version is 0.43.0.

Figure 3. The number of exposed OPA servers and their versions based on the data found on Shodan

In figure 3, we can see an example of an exposed OPA server that we have found via Shodan. Aside from being able to query the server via this page, this could serve as an entry point for command injection attacks if the user input data is not properly sanitized and validated. There is also the problem of the REST API being exposed and unauthenticated.

Figure 4. A sample of an outdated (0.29.3) and exposed OPA server was found on Shodan. As of the time of writing, the latest version of this OPA server is 0.43.

Meanwhile, this is the result of a GET request made to the /v1/policies endpoint of this exposed server:

Figure 5. Accessing the list policies (/v1/policies) endpoint from the exposed OPA server to list all the available policies

After beautifying the policies.json file generated and analyzing its content, we found some peculiar information just from the policy file itself:

• There are five rules available on this exposed OPA server with IDs that provide a clue regarding their purpose:
• "systems/ea2c8e2dbfa748608077be0d6fd45369/rules/rules.rego"
• "systems/ea2c8e2dbfa748608077be0d6fd45369/test/test.rego"
• "systems/ea2c8e2dbfa748608077be0d6fd45369/product/manager/ rules/rules.rego"
• "systems/ea2c8e2dbfa748608077be0d6fd45369/product/rules/rules.rego"
• "systems/ea2c8e2dbfa748608077be0d6fd45369/backoffice/rules/rules.rego"
• The first rule is the default, with a default return value set to false. The second rule is a testing rule.
• We have access to all the policies available on this exposed OPA server, which were accessible in raw and Abstract Syntax Tree (AST) formats.
• Some rules can reveal internal application roles such as "publisher" and "editor."
• Some base64-encoded data translate to "helpers."

Analyzing exposed OPA policies

After collecting and analyzing almost 400 policies from all these exposed OPA servers, here is what we were able to find out:

• At least 33% of these policies had some sort of sensitive information regarding the application, which included user profiles, services being used, and URLs that would trigger APIs via the Amazon Web Services (AWS) API Gateway and even some internal system URLs.
• And 91% of these policies had some sort of information about how the system should behave to bypass certain restrictions based on the policies that were implemented.
  

Figure 6. Percentage of relevant and sensitive information found on the collected OPA policies

Here are some examples of services and URLs that were found just by looking at the policies and the output they provided for unauthenticated requests:

Figure 7. Examples of services and URLs that were found just by looking at the OPA policies

With the proper request or token, an attacker could obtain even more information about these services and look for vulnerabilities or other entry points to get into an organization's systems. We highly recommend that companies currently leveraging OPA as their policy-as-code solution to ensure that they are not unwittingly exposing their APIs and policies online. In certain cases, companies could be using OPA without them realizing it; multiple providers for Kubernetes-managed services rely on OPA for policy enforcement.

Keep in mind that we only queried the list policies endpoint from the REST API for ethical reasons. However, there are many other available endpoints and methods that not only list sensitive information, but also allow an attacker to edit or even delete data and objects from an exposed OPA server. Some of these are:

Create or update a policy	PUT /v1/policies/
Delete a policy	DELETE /v1/policies/
Patch a document (Data API)	PATCH /v1/data/{path:.+}
Delete a document (Data API):	DELETE /v1/data/{path:.+}

All of these can be found in the OPA REST API Documentation.

Protecting OPA servers

Primarily, OPA servers should not be exposed to the internet. Thus, it's necessary to restrict that access to avoid anyone poking around your OPA configurations via the REST API. The standard mode of OPA deployment for the authorization use case is to have OPA running on the same machine as the application asking it for decisions. This way, organizations would not need to expose OPA to the internet or the internal network, as communication is performed over the localhost interface. Furthermore, deploying OPA this way means that organizations usually won't need authentication/authorization enabled for the REST API, as only a process running on the same machine would be able to query the OPA instance. To do that, OPA can be started with "opa run --addr localhost:8181" to have it bind only to the localhost interface.

Secondly, when using a policy-as-code tool such as OPA, it is important to protect policies in a location such as a source code management (SCM) system. It is also vital to have proper access controls to monitor who can change what in those policies via features such as branch protection and code owners. With the power of the SCM system, organizations can create a more streamlined process of reviews and approvals of any changes made to these policies, making sure that whatever is in the source code is also reflected in the production OPA servers.

TLS and HTTPS

As seen on Figure 4, most of these exposed OPA servers found on Shodan were not using any sort of encryption for communication, as this is not enabled by default. To configure TLS and HTTPS, system administrators need to create a certificate and a private key, and provide the following command line flags:

• The path of the TLS certificate: --tls-cert-file=
• The path of the TLS private key: --tls-private-key-file=

For up-to-date information regarding this process, please consult the OPA documentation on TLS and HTTPS.

Authentication and authorization

By default, OPA authentication and authorization mechanisms are turned off. This is described in OPA's official documentation, and it is vital that system administrators and DevOps engineers enable these mechanisms immediately after installation.

Both mechanisms can be configured via the following command line flags according to the OPA documentation:

• Authentication: --authentication=.
  This can be bearer tokens (--authentication=token) or client TLS certificates (--authentication=tls).
  
  
• Authorization: --authorization=.
  This uses Rego policies to decide who can do what in OPA. It can be enabled by setting the --authorization=basic flag during OPA startup and providing a minimal authorization policy.

More details pertaining to this process can be found in the OPA official documentation on authentication and authorization.

Cloud security recommendations

Kubernetes is one of the most popular platforms among developers, as proven by its high adoption rate that does not show any signs of slowing down soon. With an ever-expanding userbase, Kubernetes deployments need to be kept secure from threats and risks. To do this, developers can turn to policy-as-code-tools, which can help implement controls and validate procedures in an automated manner.

Aside from diligently applying some basic housekeeping rules to keep Kubernetes clusters secure, organizations can also benefit from cloud-specific security solutions such as Trend Micro™ Hybrid Cloud Security and Trend Micro Cloud One™.

Trend Micro helps DevOps teams to build securely, ship fast, and run anywhere. The Trend Micro™ Hybrid Cloud Security solution provides powerful, streamlined, and automated security within the organization's DevOps pipeline and delivers multiple XGen™ threat defense techniques for protecting runtime physical, virtual, and cloud workloads. It is powered by the Cloud One™ platform, which provides organizations a single-pane-of-glass look at their hybrid cloud environments and real-time security through Network Security, Workload Security, Container Security, Application Security, File Storage Security, and Conformity services.

For organizations looking for runtime workload, container image, and file and object storage security as software, the Deep Security™ scans workloads and container images for malware and vulnerabilities at any interval in the development pipeline to prevent threats before they are deployed.

Trend Micro™ Cloud One™ is a security services platform for cloud builders. It provides automated protection for cloud migration, cloud-native application development, and cloud operational excellence. It also helps identify and resolve security issues sooner and improves delivery time for DevOps teams. It includes the following:

• Workload Security: runtime protection for workloads
• Container Security: automated container image and registry scanning
• File Storage Security: security for cloud file and object storage services
• Network Security: cloud network layer IPS security
• Application Security: security for serverless functions, APIs, and applications