What is the use of robert.txt file ?

Hello friends,


I would like to know that What is the use of robert.txt file ? tell me guys..

The robots exclusion protocol (REP) or robots.txt is a text file which webmasters create to instruct robots (especially search engine robots) how to crawl and index pages on our website.

The file which is instructing to the search engine bot not to crawl this page within our website.

robots.txt is a plain text file webmaster create to instruct search engine robots how to crawl and indexed of their website.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

Robert.txt file when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty.