Explanations About Commands of Robots.txt For Blogger

In previous post we learn how to enable custom robots.txt in blogger. So if you want to enable it please visit How to Add Custom Robots.txt File In Blogger.

Robots.txt is a simple text file in which the website or blogger owner use that file to write the commands for the search engine's crawler. Indirectly we can say that it instructs the search engine's crawler that we are the part of our site can be indexed or not. That commands are written in different coding which is only for search engine's crawler. Robots.txt should be always on root directory.

So that if you want to check that file you can use following urls :

http://www.yourdomain.com/robots.txt 

In previous post we enable custom robots.txt and added command in that like

So here is that code

            User-agent: Mediapartners-Google
            User-agent: *
            Disallow: /search?q=*
            Disallow: /*?updated-max=*
            Allow: /
            Sitemap: http://www.yourdmomain.com/feeds/posts/default?orderby=updated
        

Explanations about robots.txt commands

User-agent: Mediapartners-Google : This command is used for those blogs which are using Google AdSense if you are not using Google AdSense then remove it. In this command, we're telling the AdSense's separate robot that crawl all pages which are having AdSense Ads.

User-agent: * : Here the User-agent is calling the robot and * is for all the search engine's robots like Google, Yahoo etc.

Disallow: /search?q=* : This line tells the search engine's crawler not to crawl the search pages.

Disallow: /*?updated-max=* : This one disallows the search engine's crawler to do not index or crawl label or navigation pages.

Allow: / : This one allows to index the homepage or your blog.

Sitemap : So this last command tells the search engine's crawler to index the every new or updated post.

How Can I Add a Command For Disallowing Any Page?

So if you are interested in adding your own command then you can also do it. Here is an example for disallowing the contact page

Disallow: /p/contact-us.html 

You will need to replace /p/contact-us.html which is other part of the domain with another. It mean you just need to exclude the main domain and just type remain part of the page. Two things you should remember that add your custom command below User-agent: * and per line = per command.

Conclusion :

I hope that this article would have helped you in understanding Commands of robots.txt for blogger. Please share your knowledge if you know more about this attribute. Your feedback and constructive contributions are always welcome.