According to some scientists or visionary entrepreneurs such as Elon Musk, artificial intelligence may ultimately threaten humanity. Whether intelligent or not, robots have always fascinated and frightened humans. For any self‑respecting webmaster, interacting with ro(bots) is a very real and daily activity. On a small website (max. 1000 visitors per day), they can represent up to 80% of total traffic. In internet jargon, bots are software programs that perform tedious tasks automatically or semi‑automatically. Some, like the famous Googlebot, index web pages in search engine databases.
In the advertising world, there are bots whose sole purpose is to appear as a source of visits (referrer spam) to promote a service to the webmaster (who is of course furious to see their stats polluted). Others, more discreet, are far more harmful. All of them end up clogging your server to the detriment of legitimate human visitors. They can slow it down, distort audience measurement, steal your site’s content to duplicate it elsewhere, flood it with spam, or allow hackers to discover a vulnerability that may threaten the integrity of your site’s data (see this Moz.com article for more details on these bad bots). In short, controlling server access and configuring filter options in Google Analytics are important—but little‑known—steps in an SEO strategy.
Check your server log
The fight against bots begins with regularly reading server access logs and errors (log files). If your hosting provider does not offer these files, change hosts! Here is an example entry. It is a visit from Googlebot, whose identity can be verified using a double DNS lookup procedure recommended by Google based on the IP address shown.
monsite.ch 66.249.67.73 -- [29/Apr/2015:00:06:06 +0200] GET /services HTTP/1.1 200 7987 - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
The raw log entry provides useful information about the user, such as their IP address, the date of the visit, the page visited, the method used (here a GET, but POST *could* indicate that data was submitted), the status of the visit (code 200 means a successful visit), the size of the downloaded file, and possibly the referrer page. If the visitor is a friendly bot, it indicates so in the user agent field with a URL. This URL explains the bot’s purpose (generally essential to the survival of humanity) and how to deny it access to the site. Choices can be made depending on the nature of the site. Indexing the website of a hair salon in Geneva in Yandex (Russia) or Baidu (China) is not essential for most internet users in those countries. You can therefore add a directive in a robots.txt file at the root of the site to block these search engine bots from accessing other pages.
User-agent: Baiduspider
User-agent: Yandex
Disallow: /
You can download a list of more than 650 unwanted user agents to include in your robots.txt file.
There are tools to simplify reading server access logs. The developer of the SEO software Screaming Frog offers his own version, Log File Analyser. The free version is limited but can be useful to more easily detect the troublemaker hammering your site. The free and open‑source software goaccess can generate a beautiful HTML statistical report from a server access log. However, you must be comfortable with the command line.
What to do with the bad actors ?
Of course, malicious bots couldn’t care less about your robots.txt file. You must block their access to the server more radically based on the signature they leave in the log file. On common Apache‑type servers, the webmaster creates a firewall by writing directives in a file named .htaccess, which modifies server behavior at the directory level. It is possible, for example, to reject malicious users based on criteria such as their IP address, user agent (software), origin, or visit URL. In general, unwanted users are easy to spot because they try to add parameters to URLs to find backdoors, configuration files, old backups, or inject malicious code, often causing 403 or 404 errors. If the technical aspect is daunting or you are a beginner, I strongly recommend using a security tool such as Akeeba Admin Tools Pro, which writes the .htaccess directives for you, performs security tests, provides statistics on blocked access, and gives useful information to improve your site’s overall security (available for Joomla and WordPress). There are also online lists of bad bots that you can copy into your .htaccess file. When IP addresses used by bad bots are publicly reported on sites such as abuseipdb.com or Project Honey Pot, the data can then be used by websites to proactively block these bad actors.
Since 2024 – much like the villains of the second season – a new category of bots has emerged: autonomous AI agents. Unlike traditional scripts, these agents are capable of interpreting a web page, making decisions, adjusting their behaviour, and even imitating human users to bypass protections. Some are legitimate (automated testing tools, navigation assistants, AI‑powered auditing tools), but others are used to probe vulnerabilities, scrape content, or generate artificial traffic that is difficult to distinguish from real visits. Detecting them requires more advanced techniques, such as behavioural analysis, network origin verification (AWS, Hetzner, OVH), or specialised security solutions able to identify activity patterns specific to AI agents (Cloudflare Bot Management).
How to filter bots in Google Analytics?
Unlike server logs, which contain raw server data, Google Analytics analyzes site traffic using JavaScript code embedded in your pages. The data is therefore processed on Google’s servers, and visits from the most well‑known bots are automatically filtered out in the current version, Google Analytics 4.
So now you know what’s happening behind the scenes of your website. If you need help improving its security even further, feel free to contact me.
