Apache
How to block all user agents (crawlers, bots) using .htaccess
Published
1 min read
The below will serve a 403 Forbidden if any request is from any of the stated user-agent strings.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider|archive.org|binlar|casper|checkpriv|choppy|clshttp|cmsworld|diavol|dotbot|extract|feedfinder|flicky|g00g1e|harvest|heritrix|httrack|kmccrew|loader|mi
ner|nikto|nutch|planetwork|postrank|purebot|pycurl|python|seekerspider|siclab|skygrid|sqlmap|sucker|turnit|vikspider|winhttp|xxxyy|youda|zmeu|zune) [NC]
RewriteRule .* - [R=403,L]
How to test it using browser Chrome
- Open Network tab
- Select User Agent - Google Bot
