BaiDuSpider Baidu Spider consumes traffic, robots.txt settings

Published: 2012-10-25

Note: The following BaiDuSpider equals Baidu spider, Googlebot equals google spider, Sogou Spider equals Sogou spider

I. Problem

For some reasons, search engine spiders crawling the website occupy a large amount of traffic, as shown in the figure:

Many spider crawling records can also be seen in the website access log.

If it is normal search engine spider access, it is not recommended to ban spiders, otherwise the website's inclusion and ranking in Baidu and other search engines will be lost, causing losses such as customer loss. You can prioritize upgrading the virtual hosting model to get more traffic or upgrade to a cloud server (unlimited traffic).

II. Understanding and Learning

We know that search engines all comply with the Internet robots protocol, and restrictions can be made through robots.txt. First, some examples of robots.txt are explained, then rules are set according to the above website situation. From the above screenshot, it can be seen that BaiDuSpider occupies a very large amount of traffic. First, create a robots.txt file in the website root directory.

Example 1. Prohibit all search engines from accessing any part of the website

　　User-agent: *　

Disallow: /　

Example 2. Allow all robot access (or you can also create an empty file "/robots.txt" file)　

　　User-agent: *　

　　Allow:　

Example 3. Prohibit access by a certain search engine (prohibitBaiDuSpider)

　　User-agent: BaiDuSpider　

Disallow: /　

Example 4. Allow access by a certain search engine

　　User-agent: Baiduspider　

allow:/　

Example 5. Prohibit search engine access to two directories

    User-agent: *

Disallow: /admin/

Disallow: /install/

Example 6. Only allow Baiduspider and Googlebot access

User-agent: Baiduspider
Allow: /
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /

Example 7. Prohibit Baidu search engine from crawling all images on your website

User-agent: Baiduspider
Disallow: /*.jpg$
Disallow: /*.jpeg$
Disallow: /*.gif$
Disallow: /*.png$
Disallow: /*.bmp$

III. Problem Solving (Rule Setting)

Based on the above examples, after analyzing the website logs, it is mainly Baidu crawling images that occupies traffic. There are also two directories that you do not want search engines to crawl. Set the rules as follows to solve:

User-agent: *

Disallow: /admin/

Disallow: /install/

User-agent: Baiduspider
Disallow: /*.jpg$
Disallow: /*.jpeg$
Disallow: /*.gif$
Disallow: /*.png$
Disallow: /*.bmp$

Because the update of the search engine index database takes time. Although the spider has stopped visiting the web pages on your website, the web page index information already established in the Baidu search engine database may take months to clear. That is to say, after setting restrictions, spider crawling will still be seen in the logs, and it will gradually decrease until it fully takes effect. This problem will continue for a period of time. If you need to block it as soon as possible, visit the following help center to make a complaint, and the search engine will handle it relatively quickly.

If it is ineffective after setting (or some spiders do not comply with the robots protocol, you can use: Use pseudo-static rules to intercept spider access http://www.west.cn/faq/list.asp?unid=662　）

IV. Baiduspider How to solve bandwidth congestion caused by too many crawl requests, affecting normal website access?

Visit Baidu Webmaster Tools http://zhanzhang.baidu.com/ , register a username and log in, first add the website and pass verification.

Then go to Web Crawling > Crawl Frequency > How to solve the current crawl frequency being too high? Prompt content:

You can follow the methods below to troubleshoot and solve the problem of excessive frequency one by one:

1. If you think Baiduspider has crawled links you consider worthless, please update the website robots.txt to block crawling, and then go to the robots tool page to make it take effect.

2. If Baiduspider's crawling affects the normal access of your website, please go to the crawl frequency upper limit adjustment page to lower the crawl frequency upper limit.

3. If none of the above methods can solve the problem, please provide feedback to the feedback center.

It is recommended to use the second method and adjust the Baidu crawl frequency upper limit. The following screenshot is the "Frequency Upper Limit Adjustment Page". If access is affected, the value can be appropriately reduced.

V. Knowledge Enhancement

For a more detailed understanding of spider crawling principles and robots.txt syntax, it is recommended to read the following standard materials in detail:

Other methods: Use pseudo-static rules to intercept spider access http://www.west.cn/faq/list.asp?unid=662

BaiDuSpider Help Center http://www.baidu.com/search/spider.html

Googlebot Help Center http://support.google.com/webmasters/bin/answer.py?hl=zh-Hans&answer=182072

Sogou Spider Help Center http://www.sogou.com/docs/help/webmasters.htm

Robots.txt Detailed introduction:http://baike.baidu.com/view/1280732.htm

baidu provides standard robots.txt syntax:http://www.baidu.com/search/robots.html

If Baidu spider access has been prohibited according to the process but Baidu spiders still access, please directly report it to Baidu for investigation and handling:
http://tousu.baidu.com/webmaster/suggest#1

http://zhanzhang.baidu.com/feedback/index