For search engines to keep their listings up to date, and present the most accurate search engine results, they perform an action known as a ‘crawl’. This is essentially sending a ‘bot’ (sometimes known as a ‘spider’) out to crawl the internet. The bot will then find new pages, updated pages or pages it did not previously know to exist. The end result of the crawl is that the search engine results page is updated, and all of the pages found on the last crawl are now included. It’s simply a method of finding sites on the internet.

However, there may be some instances where you have a website page you do not want included in search engine results. For example, you may be in the process of building a page, and do not want it listed in search engine results until it is completed. In these instances, you need to use a file known as robots.txt to tell a search engine bot to ignore your chosen pages within your website.

Robots.txt is basically used as a way of telling a search engine “don’t come in here, please”. When a bot finds a robots.txt file, it will ‘read’ it and will duly ignore all the URLs contained within. Therefore pages within the Robots.txt file do not appear in search results.

It isn’t a fail-safe; robots.txt is a request for bots to ignore the page, rather than a complete block, but most bots will obey the information found within the file. When you are ready for the page to be included in a search engine, you simply modify your robots.txt file and remove the URL of the designated page.

Share and Enjoy:
  • Add to favorites
  • Print
  • email
  • Facebook
  • Twitter
  • LinkedIn
  • MySpace
  • Google Bookmarks
  • Google Buzz
  • Yahoo! Buzz
  • del.icio.us
  • Digg
  • Diigo
  • MisterWong
  • Netvibes
  • Reddit
  • StumbleUpon
  • Technorati
  • Ping.fm
  • Plurk

You May Also Like:

One Response to “What Is Robots.txt?”

Leave a Reply





*

EnglishFrenchGermanItalianPortugueseRussianSpanish
Social Media Marketing Platform - Social Media Marketing Made Easy! - We provide fully-managed Social Media Management Services plus free articles knowledgebase & blog - Social Media Marketing for Business,Social Media Marketing Services Heathmont Victoria Australia
Social Media Marketing Platform - Social Media Marketing Made Easy! - We provide fully-managed Social Media Management Services plus free articles knowledgebase & blog - Social Media Marketing for Business,Social Media Marketing Services Heathmont Victoria Australia
Social Media Marketing Platform - Social Media Marketing Made Easy! - We provide fully-managed Social Media Management Services plus free articles knowledgebase & blog - Social Media Marketing for Business,Social Media Marketing Services Heathmont Victoria Australia

SEO-Land.com
(C) 2014 Business-In-Site Pty Ltd / SEO-Land.com. All Rights Reserved.
Disclaimer: The content provided on this website is for entertainment purposes only. and is not intended to be a replacement for professional business or legal advice. All images and content (C) the original authors.