A robots.txt file is a simple text file that website owners can use to tell search engine bots(crawlers) which pages or sections of their site should not be crawled. It is important to note that the robots.txt file is not a guarantee that a website will not be crawled or indexed by search engines, but it is a strong indication that certain pages should not be accessed.
How robots.txt works
The way robots.txt works is simple: web crawlers or “bots” that belong to search engines such as Google, Bing, and Yahoo, will look for a file named “robots.txt” at the root of a website, when they visit it. If the file is present, the bots will read the instructions provided in the file and will follow them while crawling the website. If the file is not present, the bots will assume that they are free to crawl the entire website.
Lets say your domain name is mywebsite.com, then the robots.txt URL should be:
http://yourdomain.com/robots.txtThe instructions provided in the robots.txt file are in the form of a set of “User-agent” and “Disallow” or “Allow” directives.
User-agent directive in robots.txt file
User-agent is a useful feature when it comes to managing access to your website by various web crawlers. With hundreds of crawlers potentially trying to access your site, it can be beneficial to set specific boundaries for each of them based on their intentions. User-agent allows you to do this by providing a way to identify the specific crawler and apply different instructions for each.
You can also use User-agent to target specific web crawlers and give them different instructions, it will depend on the crawler capabilities and behavior. Also some crawlers have their own name and format, for example Googlebot-Image, Googlebot-News, Googlebot-Video etc. You can find the user-agent name of each crawler on their official website.
Also, you can use the wildcard * to apply the instructions to all web crawlers that visit your site.
Disallow: directive
The “Disallow” directive is used to specify which pages or sections of a website should not be accessed by web crawlers.
User-agent: *
Disallow: /adminThe above example will block all URLs whose path starts with “/admin”:
http://yourdomain.com/admin
http://yourdomain.com/admin?test=0
http://yourdomain.com/admin/somethings
http://yourdomain.com/admin-example-page-keep-them-out-of-search-resultsAllow: directive
The Allow directive is used to specify which parts of a website should be accessible to web crawlers or search engine bots.
User-agent: *
Allow: /some-directory/important-page
Disallow: /some-directory/Above example will block the following URLs:
http://yourdomain.com/some-directory/
http://yourdomain.com/some-directory/everything-blocked-butBut it will not block any of the following:
http://yourdomain.com/some-directory/important-page
http://yourdomain.com/some-directory/important-page-its-someting
http://yourdomain.com/some-directory/important-page/anypage-hereSitemap directive
The Sitemap directive can be used to specify the location of a sitemap for a website. A sitemap is an XML file that lists all of the URLs on a website and provides information about each URL, such as when it was last updated. This can be useful for search engines when they are crawling a website, as it allows them to find all of the URLs on the site more easily.
Example:
User-agent: *
Sitemap: http://yourdomain.com/sitemap.xmlCrawl-delay: directive
The Crawl-delay directive can be used to specify the number of seconds that a web crawler should wait between requests to a website. This can be useful for preventing a website from being overwhelmed by too many requests from a single crawler.
Note: This directive is not a standard and not all web crawlers support it. In particular, Google does not support it, instead, it uses different methods to control the crawling rate, such as setting the crawl rate in the Google Search Console.
Example:
User-agent: *
Crawl-delay: 2Wildcards “*” asterisk in robots.txt
Wildcards can be used to specify a pattern of URLs that should be blocked or allowed for web crawlers. Wildcards can be used in both the Disallow and Allow directives. For Example
Disallow: /names/*/detailsBelow are the URLs that will be blocked from above directive
http://yourdomain.com/names/ravi/details
http://yourdomain.com/names/rohit/account/details
http://yourdomain.com/names/aarush/details-about-something
http://yourdomain.com/names/varma/search?q=/detailsEnd-of-string operator “$” (Dollar sign)
The dollar sign $ can be used to indicate the end of a URL. This can be useful in cases where you want to block a specific file type or extension.
User-agent: *
Disallow: /junk-page$
Disallow: /*.pdf$From the above example any URLs ending with pdf and junk-page will be blocked
But it will not block any of the following:
http://yourdomain.com/junk-page-and-how-to-avoid-creating-them
http://yourdomain.com/junk-page/
http://yourdomain.com/junk-page?a=bWhat if you want to block all URLs that contain a dollar sign?
http://yourdomain.com/store?price=$50The following will not work:
Disallow: /*$This directive will actually block everything on your website. To avoid this mistake, an extra asterisk should be placed after the dollar sign.
Disallow: /*$*Common Robots.txt Configuration Mistakes
Not placing the robots.txt file in the correct location
Placing the robots.txt file anywhere other than the site root will result in it being ignored by search engines. If you do not have access to the site root, you can block pages using alternative methods such as robots meta tags or by using the X-Robots-Tag header in the .htaccess file (or equivalent)
Blocking subdomains in robots.txt
Trying to target specific subdomains using robots.txt is a common mistake. Robots.txt is only applicable to the specific domain it is placed on, and will not affect any subdomains. To block specific subdomains, you will need to create a separate robots.txt file for each subdomain and place it in the root directory of that subdomain.
Case Consistency in Robots.txt
The robots.txt standard is case-sensitive, meaning that “User-agent” and “user-agent” would be interpreted as different commands. Similarly, “Disallow” and “disallow” would also be treated as different commands.
Forgetting the user-agent line
The “User-agent” line tells search engines which crawlers the rules in the file apply to. Without this line, search engines will not know which rules to follow, and may ignore the entire file.
How to test robots.txt file?
Testing the robots.txt file can be done with the use of web-based tools like Google Search Console and Bing Webmaster Tools, which allows you to enter the URL you want to verify and see if it’s allowed or disallowed. If you have technical knowledge, you can also use Google’s open-source robots.txt library to test the file locally on your computer.
Meta Robots Tag
Robots.txt is not the sole method of communicating with web crawlers. Alternative methods include using the Meta Robots Tag and X-Robots-Tag headers to specify crawling instructions for specific pages or sections of a website.
Learn more about Meta Robots Tag
