apache, iis6, ii7 independent IP host blocking and intercepting spider crawling (applicable to VPS cloud host servers)

If it is a normal search engine spider visit, it is not recommended to block the spider, otherwise the website's indexing and ranking in Baidu and other search engines will be lost, causing losses such as customer loss. You can prioritize upgrading the virtual hosting model to obtain more traffic or upgrade tocloud server (unlimited traffic).For more details, please visit: http://www.west.cn/faq/list.asp?unid=626 

1. Use the website management assistant environment:http://www.west.cn/faq/list.asp?unid=650 Refer to this description to enable and set up the pseudo-static component

2.  windows2003+iis manual website building environment:http://www.west.cn/faq/list.asp?unid=639 Refer to this description to load the pseudo-static component                 

3.  Then configure according to the following system rules in the configuration file

Under Linux rule file .htaccess (manually create the .htaccess file in the site root directory)

<IfModule mod_rewrite.c>
RewriteEngine On
#Block spider
RewriteCond %{HTTP_USER_AGENT}   "Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu|^$"   [NC]
RewriteRule !(^robots\.txt$) - [F]
</IfModule>

Under windows2003 rule file httpd.conf 

#Block spider
RewriteCond %{HTTP_USER_AGENT}   (Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu|^$)   [NC]
RewriteRule !(^/robots.txt$) - [F]

Under windows2008 web.config

<?xml version="1.0" encoding="UTF-8"?>
  <configuration>
      <system.webServer>
       <rewrite>  
         <rules>         
<rule name="Block spider">
      <match url="(^robots.txt$)"   ignoreCase="false" negate="true" />
      <conditions>
        <add   input="{HTTP_USER_AGENT}"   pattern="Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|curl|perl|Python|Wget|Xenu|ZmEu|^$"   ignoreCase="true" />
      </conditions>
      <action   type="AbortRequest" />
</rule>
        </rules>  
        </rewrite>  
       </system.webServer>
  </configuration>

Nginx corresponding blocking rules

The code needs to be added to the server section in the corresponding site configuration file

if ($http_user_agent ~* "Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|Bytespider|Java|PhantomJS|SemrushBot|Scrapy|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|perl|Python|Wget|Xenu|ZmEu|^$"   )
{
  return 444;
}

Note: Some unknown spiders are blocked by default in the rules. To block other spiders, simply add them according to the rules.

Attached names of major spiders:

google spider: googlebot

Baidu spider: baiduspider

Baidu mobile spider: baiduboxapp

yahoo spider: slurp

alexa spider: ia_archiver

msn spider: msnbot

bing spider: bingbot

altavista spider: scooter

lycos spider: lycos_spider_(t-rex)

alltheweb spider: fast-webcrawler

inktomi spider: slurp

Youdao spider: YodaoBot and OutfoxBot

Retu spider: Adminrtspider

Sogou spider: sogou spider

SOSO spider: sosospider

360 search spider: 360spider