The Bots Are Coming – And We’re Ready!

Only 60% of the traffic hitting your website or web app comes from humans. The rest isn't from animals or aliens, it's from bots - automated programs that act like a user to perform specific tasks. Some of these bots are beneficial. Let's call them Good Bots, such as search engine crawlers that help your site get ranked, site monitoring bots that track performance, bots from partners, bots that check for copyright infringement, etc., bots that enable feeds, so details are updated. These represent about 15% of overall traffic.

Then there are the Bad Bots. These are scripts that scrape the copy and images from your site, come from click fraud on ad networks, fill forms with spam, and impersonate search crawlers, and of course, the botnets that threaten to overload the site in a DDOS attack. Roughly 25% of site traffic comes from Bad Bots, degrading performance, adding to bandwidth and infrastructure costs, compromising digital assets, and threatening cyber-attacks.

Bot management is an essential layer in the application security stack. It's vital to detect, classify then act on bot traffic appropriately. We have added a Bot Management pipeline to our NSX Advanced Load Balancer (Avi Networks), so let's take a look at how it works.

Only 60% of the traffic hitting your website or web app comes from humans. The rest isn't from animals or aliens, it's from bots - automated programs that act like a user to perform specific tasks. Some of these bots are beneficial. Let's call them Good Bots, such as search engine crawlers that help your site get ranked, site monitoring bots that track performance, bots from partners, bots that check for copyright infringement, etc., bots that enable feeds, so details are updated. These represent about 15% of overall traffic.

Then there are the Bad Bots. These are scripts that scrape the copy and images from your site, come from click fraud on ad networks, fill forms with spam, and impersonate search crawlers, and of course, the botnets that threaten to overload the site in a DDOS attack. Roughly 25% of site traffic comes from Bad Bots, degrading performance, adding to bandwidth and infrastructure costs, compromising digital assets, and threatening cyber-attacks.

Bot management is an essential layer in the application security stack. It's vital to detect, classify then act on bot traffic appropriately. We have added a Bot Management pipeline to our NSX Advanced Load Balancer (Avi Networks), so let's take a look at how it works.

Bot Detection

First, we need to determine whether incoming traffic is from a human or a bot. The Bot Detection Framework examines characteristics such as the IP reputation, User Agent, and Origin Network (Autonomous System Number) to determine whether the traffic is from a person or a bot.

The Bot Detection Framework can be customized and is modular to allow granular flexibility over how the traffic is handled by the pipeline. It also has features for intelligent consolidation of data sources to make an informed detection decision. The classifications can be customized so that each organization can describe each of its 'known bots' for appropriate management.

Customized Bot Detection

Bot Classification

Not all bots are equal - some are Good, some Bad, while others are Dangerous. The Bot Classification engine controls the security policies to govern each type.

Classifications can include:

• Good Bots - such as search engine crawlers

• Bad Bots - scrapers, click fraud bots
• Dangerous Bots - impersonating humans, botnet attackers
• Custom Bots - perhaps from a partner, as defined by a BotPolicy
• Unknown - if there is not enough information to decide
• Humans - users who will be the majority of the traffic

Bot Traffic Classifications

Just as the Bot Detection engine has fine controls over the identification of bots, the Bot Classification engine also offers specific criteria to help categorize bot traffic appropriately. Bad and Dangerous Bots will emulate the behavior of Humans and Good Bots to gain access to the web service, so these classifications can be adjusted to ensure they are allocated to the correct class. Equally, the organization does not want Custom Bots being treated as hostile and prevented from their purpose.

Now that the 40% of traffic from bots has been detected and classified, it's time to take action.

Bot Actions

The Bot Management pipeline gives organizations the ability to manage bot traffic through a range of Actions actively. These include:

• Permit - allow the traffic to pass through to the next layer in the application security stack
• Deny - drop or close the connection
• Rate-limit - allow the connection up to a specific threshold
• Custom response - perhaps a certain times of day, or if other conditions are met

Several actions can be combined for specific bot classifications. The actions are determined by the overall HTTP Security Policy to fit within a broader security posture. Equally, a DataScript or the WAF can govern actions, helping the organization respond to bot traffic within a holistic security framework.

Security Rules and Actions

What's next?

Bot Management is a crucial layer in the application security stack. It is fully integrated into the NSX Advanced Load Balancer that also includes a WAF, Application Rate Limiting, DDoS Protection, AV Malware Protection, User Authentication, Encryption plus L3/4 and L7 ACLs. In addition, of course, it works across multi- and hybrid-cloud environments, including on-premises, private clouds, and public clouds, just like the rest of the application services platform. Bot Management is likely to be an ongoing project as new bot threats emerge, so we will be updating our capabilities constantly. You can get a sneak preview demo of our current capabilities here or learn more about our approach to the Application Security Stack on the blog here.

The bots may be coming - but with Bot Management: We're ready! Are you?

Register today and join us live on December 9, 2021 to learn how to strengthen your application security with bot management to detect bot traffic, determine its intent, and mitigate bad bots to optimize customer experience, protect digital assets, and prevent online fraud. Webinar available on-demand at same link after December 11, 2021

Public link

Please use the above public link if you want to share this noodl on another website.