How to Block Malicious Bots and Scrapers from WordPress

The Hidden Tax on Your Server

Did you know that over 30% of all internet traffic consists of automated bots? While some bots are helpful (like Googlebot crawling your site for SEO), "bad bots" are a massive problem. They scrape your content for spam sites, brute-force your login pages, probe for vulnerable plugins, and consume massive amounts of server CPU.

Identifying Bad Bots

You can identify bad bots by reviewing your server access logs. Look for:

• High request rates from a single IP address.
• User-Agents that are blank, outdated, or clearly malicious (e.g., `Python-urllib`, `AhrefsBot` if you don't want SEO crawlers, or `SemrushBot`).
• Repeated 404 errors requesting known vulnerable plugin files (e.g., `/wp-content/plugins/revslider/`).

How to Stop Them

1. Robots.txt (The Polite Way)

You can ask polite bots to leave by editing your robots.txt file in your root directory:

User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /

Note: Malicious hackers completely ignore robots.txt. This only stops polite commercial scrapers.

2. Blocking via .htaccess

To forcefully block bots, you can deny them by User-Agent or IP address in your .htaccess file.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|python|wget|curl).* [NC]
RewriteRule .* - [F,L]
</IfModule>

This returns a 403 Forbidden error to common script agents.

3. Dynamic Protection with a WAF

The problem with `.htaccess` rules is that modern bots rotate their IP addresses and spoof their User-Agents to look like normal Chrome browsers. Static rules cannot stop them.

You need behavioral analysis. Nexura Security's Web Application Firewall monitors request rates and behavior. If an IP requests 50 pages in 5 seconds, Nexura instantly recognizes it as a bot and blocks the IP temporarily, regardless of what User-Agent it claims to be. This ensures your human visitors get a fast website, while the bots hit a brick wall.