Robots.txt Validator


See how to use Robots.txt Validator

The Robots Exclusion Protocol, commonly referred to as /robots.txt, is used to give direction and information about a given website to web robots. When a robots visits a specific website, the first thing they do is find the robots.txt file to identify which pages, if any, are disallowed.

It’s important to note that regardless of any direction given, robots can choose to ignore the file and /robots.txt files are public.

Having been first developed in 1994, the original document has undergone some changes, resulting in a revised Method of Robots Control in 1997. There is currently no effort being made to develop further /robots.txt; however, unofficial efforts to extend robots exclusion mechanisms had been discussed in 2008.

This tool can help identify errors that may exist within your current /robots.txt file. It also lists the pages that you’ve specified to be disallowed.


Feedback

 

Users comments:

Login or register to post comments and rate

Not rated yet.

doctor william
2017-08-15
My name is Dr William from KENYA. I am thankful to all of you out there that have been posting my email on the Internet telling people what I have been able to do for you I thank you all for the testimonies . And I also want to inform those who have not yet been informed If you have any kind of problem don't waste time to contact me because a problem been shared is a problem also solved. life is just one there is no second chance, MY SPECIALIZATION INCLUDE TREATMENT ON (1) asthma, (2)cancer, (3)Diabetic (4) Gonorrhea, (5) Barrenness, (6)Impotence, (7) High Blood Pressure, (8) Epilepsy, (9) Hiv aids etc I ALSO SPECIALIZE ON (1) Do you need a spell caster to bring back my ex husband/wife/lover, inbox me today, so that we can work together and bring he/her back, so that you both can live happily ever after.. (2) to stop Bad Dreams Spell (3) If You want to be promoted in your office. (4) You want women/men to run after you. (5) Pregnancy Spell. (6) Rich Spell. (7) You want to be in control in. everything. (9) Good luck Spell. (10) Love spell. (11) If you want your wife or your husband to be yours forever. (12) Spell to break barrenness. (13) Spell to be Famous (14) spell to cure any kind of sickness Inbox me today on doctorwilliam06@gmail.com,all your problems are solve or add our mother queen of healing on Facebook (Angela winfred)
sumit singh
2017-01-17
Not to very good
Anonymous
2016-01-25
http://anh-sex-lon-con-trinh.sextgem.com