August 16, 2023

How to block all user agents (crawlers, bots) using .htaccess

The below will serve a 403 Forbidden if any request is from any of the stated user-agent strings.

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider|archive.org|binlar|casper|checkpriv|choppy|clshttp|cmsworld|diavol|dotbot|extract|feedfinder|flicky|g00g1e|harvest|heritrix|httrack|kmccrew|loader|mi
ner|nikto|nutch|planetwork|postrank|purebot|pycurl|python|seekerspider|siclab|skygrid|sqlmap|sucker|turnit|vikspider|winhttp|xxxyy|youda|zmeu|zune) [NC]
RewriteRule .* - [R=403,L]

How to test it using browser Chrome

  1. Open Network tab
  2. Select User Agent - Google Bot screenshot-at-16-56-57.png

Not sure which platform or technology to use?

We can turn different applications and technologies into a high-performance ecosystem that helps your business grow.