Methods to prevent search engines from indexing

1. What is a robots.txt file?
   Search engines use spider programs to automatically visit webpages on the Internet and obtain webpage information. When a spider visits a website, it will first check whether there is a plain text file called robots.txt under the root domain of the website. This file is used to specify the crawling scope of the spider on your website. You can create a robots.txt in your website and declare in the file the parts of the website that you do not want search engines to index, or specify that search engines should only index specific parts.
   
   Please note that you only need to use a robots.txt file when your website contains content that you do not want search engines to index. If you want search engines to index all content on your website, do not create a robots.txt file.
   
   

    

2. Where should the robots.txt file be placed?
   The robots.txt file should be placed in the root directory of the website. For example, when a spider visits a website (such as http://www.abc.com), it will first check whether the file http://www.abc.com/robots.txt exists in that website. If the Spider finds this file, it will determine the scope of its access permissions according to the content of this file.
   
   Website URL Corresponding robots.txt URL http://www.w3.org/ http://www.w3.org/robots.txt http://www.w3.org:80/ http://www.w3.org:80/robots.txt http://www.w3.org:1234/ http://www.w3.org:1234/robots.txt http://w3.org/ http://w3.org/robots.txt
   

    

3. I have set in robots.txt that Baidu should not index my website content, why does it still appear in Baidu search results?
   If other websites link to webpages that you have set as prohibited from indexing in your robots.txt file, then these webpages may still appear in Baidu search results. However, the content on your webpages will not be crawled, indexed, or displayed. What is displayed in Baidu search results is only other websites' descriptions of your related webpages.
   
   

    

4. Prevent search engines from following links on a webpage, while only indexing the webpage
   If you do not want search engines to follow links on this webpage and do not pass link weight, place this meta tag in the <HEAD> section of the webpage:
   <meta name="robots" content="nofollow">
   If you do not want Baidu to follow a specific link, Baidu also supports more precise control. Write this tag directly on a link:
   <a href="signin.php" rel="nofollow">sign in</a>
   To allow other search engines to follow links, but only prevent Baidu from following links on your webpage, place this meta tag in the <HEAD> section of the webpage:
   <meta name="Baiduspider" content="nofollow">
   

    

5. Prevent search engines from displaying webpage snapshots in search results, while only indexing the webpage
   To prevent all search engines from displaying snapshots of your website, place this meta tag in the <HEAD> section of the webpage:
   <meta name="robots" content="noarchive">
   To allow other search engines to display snapshots, but only prevent Baidu from displaying them, use the following tag:
   <meta name="Baiduspider" content="noarchive">
   Note: This tag only prevents Baidu from displaying the snapshot of the webpage. Baidu will continue to index the webpage and display the webpage summary in search results.
   
   

    

6. I want to prevent Baidu Image Search from indexing certain images. How should I set this up?
   Prohibiting Baiduspider from crawling all images on a website, or prohibiting or allowing Baiduspider to crawl image files of a specific format on a website, can be achieved by setting robots. Please refer to Examples 10, 11, and 12 in "robots.txt file usage examples".
   
   

    

7. Format of the robots.txt file
   The "robots.txt" file contains one or more records. These records are separated by blank lines (with CR, CR/NL, or NL as terminators). The format of each record is as follows:
   "<field>:<optionalspace><value><optionalspace>".
   
   In this file, # can be used for comments, and the specific usage is the same as the convention in UNIX. Records in this file usually start with one or more User-agent lines, followed by several Disallow and Allow lines. Details are as follows:
   
   User-agent:
   　　 The value of this item is used to describe the name of the search engine robot. In the "robots.txt" file, if there are multiple User-agent records, it means multiple robots will be restricted by "robots.txt". For this file, there must be at least one User-agent record. If the value of this item is set to *, it is valid for any robot. In the "robots.txt" file, there can only be one record such as "User-agent:*". If you add "User- agent:SomeBot" and several Disallow and Allow lines in the "robots.txt" file, then the robot named "SomeBot" is only restricted by the Disallow and Allow lines after "User-agent:SomeBot".
   
   Disallow:
   　　 The value of this item is used to describe a group of URLs that you do not want accessed. This value can be a complete path or a non-empty prefix of a path. URLs beginning with the value of the Disallow item will not be visited by the robot. For example, "Disallow:/help" prohibits the robot from accessing /help.html, /helpabc.html, /help /index.html, while "Disallow:/help/" allows the robot to access /help.html and /helpabc.html, but not /help/index.html. "Disallow:" indicates that the robot is allowed to access all URLs of the website. In the "/robots.txt" file, there must be at least one Disallow record. If "/robots.txt" does not exist or is an empty file, then the website is open to all search engine robots.
   
   Allow:
   　　 The value of this item is used to describe a group of URLs that you want accessed. Similar to the Disallow item, this value can be a complete path or a prefix of a path. URLs beginning with the value of the Allow item are allowed to be accessed by the robot. For example, "Allow:/hibaidu" allows the robot to access /hibaidu.htm, /hibaiducom.html, /hibaidu/com.html. All URLs of a website are allowed by default, so Allow is usually used together with Disallow to allow access to some webpages while prohibiting access to all other URLs.
   
   Special note: the order of Disallow and Allow lines matters. The robot will determine whether to access a certain URL according to the first successfully matched Allow or Disallow line.
   
   Using "*" and "$":
   Baiduspider supports using the wildcards "*" and "$" for fuzzy matching of URLs.
   
   　　 "$" matches the end-of-line character.
   　　 "*" matches 0 or more arbitrary characters.
   
   

    

8. URL matching examples
   Value of Allow or Disallow URL Match result/tmp/tmpyes/tmp/tmp.htmlyes/tmp/tmp/a.htmlyes/tmp//tmpno/tmp//tmphohono/tmp//tmp/a.htmlyes/Hello*/Hello.htmlyes/He*lo/Hello,loloyes/Heap*lo/Hello,lolonohtml$/tmpa.htmlyes/a.html$/a.htmlyeshtm$/a.htmlno
   

    

9. Examples of robots.txt file usage Example 1. Prohibit all search engines from accessing any part of the website Download this robots.txt fileUser-agent: * Disallow: /Example 2. Allow all robots to access (or you can also create an empty file "/robots.txt")User-agent: *Disallow: orUser-agent: *Allow: / Example 3. Only prohibit Baiduspider from accessing your websiteUser-agent: BaiduspiderDisallow: /Example 4. Only allow Baiduspider to access your websiteUser-agent: BaiduspiderDisallow: User-agent: *Disallow: /Example 5. Prohibit spider from accessing specific directories　　 In this example, the website has three directories with restrictions on search engine access, that is, the robot will not access these three directories. Note that each directory must be declared separately, and cannot be written as "Disallow: /cgi-bin/ /tmp/".User-agent: *Disallow: /cgi-bin/Disallow: /tmp/Disallow: /~joe/Example 6. Allow access to some URLs in specific directoriesUser-agent: *Allow: /cgi-bin/seeAllow: /tmp/hiAllow: /~joe/lookDisallow: /cgi-bin/Disallow: /tmp/Disallow: /~joe/Example 7. Use "*" to restrict URL access　　 Prohibit access to all URLs under the /cgi-bin/ directory with the suffix ".htm" (including subdirectories).User-agent: *Disallow: /cgi-bin/*.htmExample 8. Use "$" to restrict URL access　　 Only allow access to URLs with the suffix ".htm".User-agent: *Allow: .htm$Disallow: /Example 9. Prohibit access to all dynamic pages on the websiteUser-agent: *Disallow: /*?*Example 10. Prohibit Baiduspider from crawling all images on the website　　 Only allow crawling webpages, prohibit crawling any images.User-agent: BaiduspiderDisallow: /*.jpg$Disallow: /*.jpeg$Disallow: /*.gif$Disallow: /*.png$Disallow: /*.bmp$Example 11. Only allow Baiduspider to crawl webpages and .gif format images　　 Allow crawling webpages and gif format images, and do not allow crawling images of other formats.User-agent: BaiduspiderAllow: /*.gif$Disallow: /*.jpg$Disallow: /*.jpeg$ Disallow: /*.png$Disallow: /*.bmp$Example 12. Only prohibit Baiduspider from crawling .jpg format imagesUser-agent: BaiduspiderDisallow: /*.jpg$
   

    

10. robots.txt file references
    For more specific settings of the robots.txt file, please refer to the following links:
    · Web Server Administrator's Guide to the Robots Exclusion Protocol
    · HTML Author's Guide to the Robots Exclusion Protocol
    · The original 1994 protocol description, as currently deployed
    · The revised Internet-Draft specification, which is not yet completed or implemented

The above document is excerpted from "Baidu Help Center"http://www.baidu.com/search/robots.html