Understanding Traffic Bots & How to Stop Them

Web bot hits originate from machines, not humans. The click, comment, or visit you believe comes from a person actually stems from a computer program instead.

Web bots aren't intrinsically good or bad. Computers can tackle repetitive tasks very quickly, and web bots can do many things critical to your performance and tracking. But web bot hits can also cost you quite a bit of money.

Web bot hits are responsible for about 40 per cent of all internet traffic. Bots really are everywhere.

Let's dig deeper into what separates the good from the bad. We'll end with some tips you can use to both monitor and stop bot traffic.

What is a traffic bot?

Web bot hits stem from a programmed script. A developer creates a program that can click links, download images, or handle some other kind of simple task.

When a program like this is written, a developer can launch it on almost any website and wait for results with no more work required. Most bots run almost all the time.

Chances are, your website deals with at least some kind of bot. About 85 per cent of websites that analyse traffic use Google Analytics to do so. If you have this tool installed on your site, look for traffic that is:

• Bouncy. A bot needs mere seconds to look over a page. If your bounce rate is sky-high, you're likely dealing with a bot.
   
• Deep. A traffic bot will likely look over your entire site, working page by page.
   
• Foreign. If your website serves a local audience and you suddenly have plenty of visitors from far-off places, you're likely dealing with a bot. 

You may also spot traffic bots by behaviour on your site. If something you post suddenly has hundreds of hits and your comment section is full of nonsensical phrases, a bot could be at work.

Good traffic bots

You've likely written your content with humans in mind, and it can be hard to think about computers taking up all of your traffic. But some website bots can be good for your performance.

These are four examples of good bots:

1. Aggregation: These website bots gather data from multiple websites and pop them into one page. A bot like this could put your website on the first page of Google as a so-called "featured snippet." That could result in thousands of hits from real people.
    
2. Analytics: These traffic bots monitor how many people visit your site, what pages they visit, how quickly your pages load, and more. This data could help sites like Google and Bing determine the value of your content.
    
3. Copyright: If someone steals your content and you decide to take legal action, you could get up to $150,000 in damages per infringement. Copyright bots help you spot the theft quickly and prove that you've been harmed.
    
4. Search engine optimisation: SEO bots walk through each page of your website and determine what you're discussing. That helps sites like Google and Bing determine what search terms should trigger your pages.

Any or all of these tasks could be critical to your success.

Bad traffic bots

Programmers can use traffic bots to attack your site, steal your resources, or both.

These are six examples of bad bots:

1. Ad fraud: Traffic bots programmed to click your paid ads can drain your bank account. Experts say ad fraud like this costs companies about $35 billion annually.
    
2. Click fraud: Programmers can create scripts that repeatedly load a page on your website. This could make a web page seem more prominent than it really is. And if you have something embarrassing on your website (such as a press release you wish you hadn't written), it could make that page very visible.
    
3. DDoS. Web bots can attempt to load your website repeatedly, and all of that traffic could overload your server and make your site unavailable. Sometimes, hackers ask for a payment before they will turn off their bots.
    
4. Scraping: Do you publish email addresses or contact information on your website? Scraping bots can pull that information and sell it to the highest bidder.
    
5. Spam: Website bots can fill out forms on your website. They could fill your inbox with nonsense messages or publish strange responses in your comments section.
    
6. Vulnerability: A programmer creates a web bot to crawl your pages and look for security problems. A report moves back to a hacker that can attack you.

Any or all of these bad bots could cost your company a lot of money.

How to stop the bad bots

It's not smart to keep all bots away from your website. You need some of the benefits they offer. But you should do all you can to keep bad bots away.

Try these three steps:

1. Adjust your website settings. A tiny bit of code (a robots.txt file) tells bots what they should and should not crawl on your site. Use it to block some pages from inspection.

   But know that only good bots play by the rules. The bad bots will likely do whatever they want, regardless of whether you like it or not.
    

2. Hire a programmer. Ask an expert to walk through your reports and spot IP addresses responsible for bot traffic. Block them manually with a filtering tool.
    
3. Use a tool. Bot management solutions use code to stop bad bots from causing chaos. These tools use machine learning, so they get more effective the longer you use them.

At Okta, we offer an excellent bot management solution. Our integration with Perimeter X keeps bad bots from threatening your business. Find out how it works.

References

Usage Statistics and Market Share of Google Analytics for Websites. W3 Techs.

Bad Bots Now Make Up 20 Percent of Web Traffic. (April 2019). ZDNet.

Copyright Infringement. Digital Media Law Project.

CHEQ Report: Cost of Digital Ad Fraud Will Rise to $35 Billion Globally in 2020. (November 2020). Martech Series.