What is Robot.txt? Robot Exclusion Standard or Robot Exclusion Protocol provides information to search engine spiders on the directories that have to be skipped or disallowed in your website. Robots.txt protocol is as important as site structure, site content, search engine friendliness and Meta descriptions.