What is a robots.txt file?
Robots.txt is a text file created by webmasters to tell search engine bots how to crawl pages on their websites. Suppose, if you want to prevent search engines from crawling and indexing your “thank you” page then you can specify a disallow command in your robots.txt file in just a single line. In this way, […]