Bot Management: Safeguarding businesses against automated attacks

Different bot developers create bots for a variety of purposes, making it challenging to assess the impact of a particular bot. Businesses and Bot Management solutions need to understand the nature of each type of bot, how to leverage the potential of good bots, and how they can impact business operations and information systems.

Good bots

Search engine crawlers: Crawlers like Googlebot, Bingbot, and spider bots from Yahoo!, Yandex, and other search engines can benefit a business's website by allowing it to be indexed and ranked on the respective search engines.

Partner bots: Depending on the business model, a company may allow bots from third-party retailers and services to crawl their website. For example, SEO tools can deploy bots on a website to perform SEO analysis or automate website optimization. The impact of good bots can vary, but they usually adhere to the guidelines and policies set in the website's robots.txt file, so businesses can control them if they consume excessive resources.

Bad bots

Bad bots are any bots programmed to operate in a "malicious" manner, such as bots that scrape data without authorization or those that perform DDoS attacks to overload server resources. To combat them, businesses need a robust Bot Management solution.

See More:

• What are bad bots and how to stop bad bot traffic.
• What is DDoS attack? The best way to prevent DDoS.

Alarming increase in bad bots

Cybersecurity threats are becoming increasingly sophisticated with the rise of attacks using malicious bots. These bots are often IoT devices or compromised computers, remotely controlled to perform various malicious activities such as account takeover (ATO), web scraping, distributed denial-of-service (DDoS) attacks, and other complex attacks.

Botnets are growing at an alarming rate globally

Nearly half (49.6%) of all internet traffic in 2023 came from bots - a 2% increase from the previous year and the highest level recorded since 2023. Bad bots have continued their upward trend for five consecutive years, with the proportion of web traffic related to malicious bots reaching 32% in 2023, up from 30.2% in 2022, while traffic from real users has declined to 50.4%.

Automated attacks - using bots - are causing billions of dollars in damages to organizations each year with attacks targeting Web, Apps, and APIs.

Some bot trends in 2023 include:

• The average global bad bot rate reached 32%: Ireland (71%), Germany (67.5%), and Mexico (42.8%) were the countries with the highest levels of bad bot traffic in 2023. The United States also saw a slight increase to 35.4% compared to 32.1% in 2022.
• Leveraging AI to amplify bot traffic: The rapid advancement of new-generation artificial intelligence (AI), particularly large language models (LLMs), has facilitated a significant surge in bot activity. Specifically, simple bots have increased from 33.4% to 39.6% in 2023. The easy accessibility of AI tools and bot development platforms has enabled users, even those without extensive programming knowledge, to create automated bots for data scraping, repetitive tasks, and even malicious purposes such as DDoS attacks.

See more: AI technology: Opportunities and challenges in DDoS mitigation.

• Account takeovers pose a major risk: The risk of account takeovers (ATO) is on the rise, with a 10% increase in 2023. ATO attacks are primarily targeting API endpoints, accounting for 44% of all attacks, a significant increase from 35% the previous year. This indicates that cybercriminals are increasingly targeting APIs to infiltrate systems and steal data from organizations, particularly in the financial services (36.8%), travel (11.5%), and business (8%) sectors.
• APIs are prime targets for attacks: Cybercriminals are actively exploiting automated bots to exploit business logic vulnerabilities in APIs, accounting for 17% of all API attacks. These vulnerabilities, often caused by design or implementation flaws, allow attackers to manipulate legitimate functions to gain unauthorized access to sensitive data, causing significant harm to organizations.
• All industries face bot threats: Bot attacks have become a pervasive problem across multiple industries. The gaming industry continues to be the top target with the highest percentage of malicious bots (57.2%) for two consecutive years. The retail, travel, and financial services industries are also heavily impacted, with bot attack rates of 24.4%, 20.7%, and 15.7% respectively. Of particular concern is the rise of sophisticated bots capable of mimicking human behavior in the legal, entertainment, and financial services sectors, with rates of 75.8%, 70.8%, and 67.1% respectively.
• Malicious bot traffic originating from residential ISPs has increased by 25.8%: Initially, techniques to evade detection of malicious bots primarily relied on mimicking regular user browsers. However, in 2023, the landscape has shifted significantly with 44.8% of malicious bot traffic masquerading as mobile devices, up from 28.1% just five years ago. Sophisticated attackers have combined the use of mobile browsers **with Residential ISP or mobile networks. Residential proxies allow attackers to disguise the origin of their traffic, making it appear as a legitimate Residential IP assigned by an ISP. This helps bots evade detection by security systems (Source: Statista).

How bad bots harm businesses?

The increasing number of bad bots is causing significant negative impacts on businesses, including the following key aspects:

• Negative impact on SEO: Web scraping bots can severely harm SEO by copying and extracting unique content. The presence of multiple identical content versions online not only infringes on copyright but also makes it difficult for search engines to identify the original website, leading to a decrease in the website's ranking and reputation.
• Eroding customer trust: By sending spam emails containing malware, creating fake product reviews, or manipulating public opinion on social media, bots erode customer trust in brands. These actions not only reduce the number of potential customers but also create unnecessary controversies, damaging the company's image.
• Distorting analytical data: Botnet attacks not only threaten the system's operational capabilities but also severely disrupt analytical data. Specifically, Distributed Denial of Service (DDoS) attacks launched by botnets can overload servers, causing service disruptions and distorting actual traffic data. Additionally, bots can generate large amounts of fake traffic, create fake accounts, and cause "cart abandonment", altering customer behavior data. This leads to business decisions based on inaccurate data, wasting resources and reducing marketing effectiveness.
• Decreasing advertising revenue: Bots can perform click fraud by automatically clicking on ads. This distorts reporting data for advertisers, causing companies to pay for fake clicks. More seriously, companies do not generate revenue from these fake "customers". Click fraud can also be used to increase the advertising costs of competitors.
• Loss of revenue: Malicious bots can negatively impact profits through various means such as: unresponsive or flagged websites, visitors being redirected to competitors, sales staff spending time on fake leads or potential customers, increased advertising costs due to fake clicks or making wrong business decisions based on inaccurate data.

Defining Bot Management

Bot Management is a set of processes, tools, and techniques to detect, classify, and respond to automated traffic (bots) on web systems and applications. This process helps differentiate between beneficial bots (e.g., search bots) and malicious bots (e.g., attack bots and illegal data scraping bots).

Why is Bot Management important?

Given the rapid increase in large-scale bot attacks, it is impossible to manually prevent them. The emergence of Bot Management solutions has provided an effective solution to this challenging problem.

Bot Management model to prevent malicious bots

There are three core benefits of having a Bot Management solution:

1. Real-time detection and identification of bot activity: An effective Bot Management suite must be able to distinguish between good and bad bots, as well as identify bad bots that are impersonating users. The core idea of bot management is to leverage the benefits of good bots while limiting or preventing the activities of malicious bots. Additionally, even if good bots do not have malicious intent, not all of them are beneficial to the business website - therefore, proper management of good bots is still necessary to prevent unnecessary waste of resources. Example: A business has a website selling products in the Vietnamese market. Allowing Chinese search engines like Baidu to crawl the business's website is like opening a store for foreigners to visit without the intention of shopping. This not only wastes time and resources but can also cause unnecessary trouble. A good Bot Management solution would be like an intelligent guard, able to distinguish between potential customers and those who just want to look around, thereby helping businesses focus on customers who are truly interested in the product.

2. Minimizing the negative impact of bots on website performance: Even good bots do not provide 100% benefits as they can consume resources when traffic increases. A suitable bot management solution can help businesses manage good bot traffic according to the current traffic of the website and other metrics. For example, a Bot Management solution can redirect or reduce the amount of good bots during peak hours to ensure maximum website performance and reduce bounce rates (as well as minimize revenue loss).

3. Controlling malicious bot activity: Bad bots can cause a variety of negative impacts, from stealing content, performing scalping attacks, to launching credential stuffing attacks, brute force attacks, and even DDoS attacks. However, just like applying static rules to detect bot traffic is insufficient, a bot detection software should provide optimized attack responses for each different threat. In addition to hard blocking bad bots, effective bot management solutions can use various methods such as throttling bot traffic, redirection, or providing fake information (honey pots) to control bot activity.

General introduction to VNIS

VNIS - A comprehensive Web/App/API security solution

The VNIS platform is a comprehensive Web/App/API security solution provided by VNETWORK Corporation. With over 2,300 Points of Presence (PoPs) globally, the VNIS solution offers the ability to handle traffic up to 2,600 Tbps, along with the support of a team of experts and a Security Operations Center (SOC) that is always ready to respond promptly to any attack situation, ensuring that your business website always operates stably with 100% uptime. Equipped with WAF, Anti-DDoS, Bot Management, and many other features, VNIS provides advanced security solutions, protecting against attacks from vulnerabilities and completely blocking large and sophisticated DDoS attacks to ensure the safety of your business systems.

VNIS Bot Management feature

In this article, we will specifically highlight an outstanding feature of the VNIS solution - Bot Management, a feature that allows businesses to effectively manage unwanted/malicious bot traffic to their domain. The intuitive and simple design interface makes it easy to manage the enable/disable feature. In addition, settings can be changed based on specific security requirements.

Enable/disable the feature flexibly

There are 3 configurable settings including Security level, Challenge passage, Challenge mode.

Security level

This allows businesses to set the security sensitivity to trigger a challenge. The system will monitor the number of requests per minute and then require authentication when the 'request threshold' is exceeded. The authentication request must include a unique token to pass the security check. The challenge will be reactivated if the request with the same token exceeds the request threshold. The following is a detailed configuration for the available options:

Detailed configuration for available options

Challenge passage

Set the waiting time for the next challenge if the same user/customer requests again. For example, if the challenge content is set to 5 minutes, the security threshold check will be reactivated after 5 minutes from when the same user completes the last challenge.

Challenge mode

You can select the challenge mode based on the specific situation.

• Browser-based (no delay): Will launch a JavaScript challenge to determine if the request is sent by a bot before executing the request.
• Browser-based (standard): Will launch a JavaScript challenge, redirecting the request to a waiting page for 5 seconds before executing the request.
• Human-based: Will launch a CAPTCHA challenge, redirecting the request to a verification page where the user must complete the "I'm not a robot" test before executing the request.