apache, iis rules block and intercept spider crawling

Published: 2013-06-04

If it is a normal search engine spider visit, it is not recommended to block the spider; otherwise, the website's inclusion and rankings in Baidu and other search engines will be lost, causing losses such as customer churn.

You may prioritize upgrading your virtual hosting model to get more traffic, or upgrading to acloud server (unlimited traffic). For more details, please visit:　http://www.west.cn/faq/list.asp?unid=626

Rules file under Linux .htaccess(manually create a .htaccess file in the site root directory)

<IfModule mod_rewrite.c>
RewriteEngine On
#Block spider
RewriteCond %{HTTP_USER_AGENT} "Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu|^$" [NC]
RewriteRule !(^robots\.txt$) - [F]
</IfModule>

Rules file under Windows2008, 2012 or later systems web.config (manually create a web.config file in the site root directory)

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
   <rewrite>
    <rules>
     <rule name="Block spider">
      <match url="(^robots.txt$)" ignoreCase="false" negate="true" />
      <conditions>
      <add input="{HTTP_USER_AGENT}" pattern="Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu|^$" ignoreCase="true" />
      </conditions>
       <action type="AbortRequest"/>
     </rule>
    </rules>
   </rewrite>
  </system.webServer>
</configuration>

Note: The line containing “{HTTP_USER_AGENT}” lists unknown spider names; add them as needed, separated by "|".
The rule blocks some unknown spiders by default. To block other spiders, add them according to the rule. Names of major spiders are listed below:
google spider: googlebot
Baidu spider: baiduspider
Baidu mobile spider: baiduboxapp
yahoo spider: slurp
alexa spider: ia_archiver
msn spider: msnbot
bing spider: bingbot
altavista spider: scooter
lycos spider: lycos_spider_(t-rex)
alltheweb spider: fast-webcrawler
inktomi spider: slurp
Youdao spider: YodaoBot and OutfoxBot
Retu spider: Adminrtspider
Sogou spider: sogou spider
SOSO spider: sosospider
360 search spider: 360spider

apache, iis block and restrict IP access (applicable to virtual hosting)

apache, iis rule settings for hotlink protection

Please refer to the ticket and configure according to the actual situation.

If you need our company to help with the configuration,please submit a ticket, and a fee of 100 yuan to 200 yuan will be charged according to the actual problem.