Tutorial On Robots.Txt

by Matthew Anton

30 Comments

Aug 21, 2013

What is Robot.txt? Robot Exclusion Standard or Robot Exclusion Protocol provides information to search engine spiders on the directories that have to be skipped or disallowed in your website. Robots.txt protocol is as important as site structure, site content, search engine friendliness and Meta descriptions.