Search for:

Resources

Articles

(1. Simple SEO Truths.)   |   (2. Your Guide to Top Search Engine Rankings. )  |   (3. Is Your Web Site Missing These Crucial Conversion Factors?)  |   (4. Internet Marketing Is Going To The Feeds.)  |   (5. Essentials for the Online Marketer's Toolbox.)  |   (6. Social Networks Search Engine Marketing for Web2.0)  |   (7. 25 Ways to Add Quality Content to Your Web Site)  |   (8. 10 Worst AdWords Campaign Management Mistakes)  |   (9. SEO guidelines)  |   (10. Market Research)   |   (11. Keyword Research)   |   (12. Online Sales Copy)   |   (13. Search Engine Friendly Web Design)   |   (14. Search Engine Friendly Dynamic Sites)   |   (15. Search Engine Friendly Design - Flash)   |   (16. Search Engine Friendly Design - The Splash Page)   |   (17. Search Engine Friendly Design - Tables)   |   (18. Search Engine Friendly Design - JavaScript)   |   (19. Search Engine Friendly Design - Cascading Style Sheets)   |   (20. Search Engine Friendly Design - Server Side Includes)  |   (21. Search Engine Friendly Design - Robots.txt)  |   (22. Web Site Optimisation)  |   (23. Web Site Usability Checklist)

Search Engine Friendly Design - Robots.txt

All major search engines support the robots.txt and it is the first file a spider will request when visiting a site. Therefore a robots.txt file should be part of every site. It should be used to exclude the robots from sensitive material that is not password protected, any under construction pages, test sections, js files, style sheets and the CGI-BIN.

Robots.txt files are not only for useful for stopping the engines from grabbing sensitive pages they also help to tell the engines where they should go instead. If you have a very large site the engines may spend so much time spidering your image directory, test directory or other non-essentials that they run out of time and go off to another site. This is not what you want. You want the engines to get to the important, keyword rich pages on your site. So use a robots.txt to keep them away from anything that isn't going to be beneficial to your search engine rankings.

As the robots.txt is explicit you should never “Allow” anything on the file unless that is the ONLY directory or page that you want the engines to index. As that would be a very rare occurrence it is best to primarily use the file to disallow what you don't want indexed instead. Run the robots.txt through a validator to ensure it's written correctly or it may end up doing more harm than good.

An example of a robots.txt
RobotsTxt.Org
Robots.tx Validator