Tweet

What are Robots.txt files

Written by: Jeff Foster

Article Overview: Through this tutorial we'll see what a robots.txt file is, how can you make one, what are it's uses and how you can use it on your website if you have one. Please follow this link if you would like to hire an internet consultant from WebBizIdeas.com.

What are Robots.txt files

Through this tutorial we'll see what a robots.txt file is, how can you make one, what are it's uses and how you can use it on your website if you have one. Please follow this link if you would like to hire an internet consultant from WebBizIdeas.com.

What are Robots.txt files ?

Robots.txt file is a text file (yea, the ones on notepad) that resides on your server, and controls a whole lot of features on your website (whatever platform it is built on). It's a simple text file in which there are a few lines of text, but it's very powerful that it can even decide whether your website should be shown on Google or not, what part of your website should be shown to the search engines (like Google, Yahoo and MSN).

What is a Robots.txt file technically?

In order to understand what robots.txt files are, you have to first understand what a Robot (the web one) is.

A robot - is technically a program from search engines like Google, Yahoo and MSN that are set out on the internet to do the job of finding out new websites, indexing them and gathering the right information about the website. They are sometime called "spiders", "crawlers" and even "bots".



Image by Elliance

Where do the Robots come from?

Robots are commonly set out by search engines like Google, Yahoo, MSN, Altavista, Ask.com and others. Mainly, these are web servers of the search engines, that are on the constant look out of information on the internet. And they gather information (which ultimately goes to the search engines index) by visiting new websites, gathering up new information from them, following links and calculating and analyzing a whole lot of information from them.

What do Robots do?

Robots mainly performs four types of tasks.

• Site Indexing - Which is more like taking a copy of a new website it finds and storing it in some location at the search engines servers. This is accomplished by scanning the documents on a website and mirroring them to temporary servers.


• Validates the site code - Which is more like comparing the website code to W3C standards and grading them according to accuracy.


• Link Checks - Which includes tracing all possible links (incoming and outgoing) from indexed websites, and calculating the sites grading factors such as authority, relevance etc.

Not all the search engine robots are the same, some advanced ones like that of Google are known to do more complex tasks like categorizing websites, analyzing their search engine metrics, popularity ratio etc, but generally all the robots perform the above tasks.




'The Life of a Google Robot' image by Google

What does a Robots.txt file do?

Robots.txt file gives commands to the visiting robots (on the website) to help them index and collect relevant information about the website.

It's more like the helpdesk, which will give all information, guidance and help to the visitors at an event about how to reach the venue, important places, time schedule, map etc.

The commands on the robots.txt file is completely configurable by the webmaster.

Using the right commands, a webmaster can decide everything related to search engines like what search engines are allowed into the website, what is the information available to them, what are the documents that are not available for the search engines and even pass information like how often are pages added to the website and how often should the robots visit them.

Where to spot the Robots.txt file?

The Robots.txt file is located at the root folder of your website. This is most often the _public-html or the http-docs folder. Root folder means the top most directory on the website that is accessible to the public. 
It is critical to place the Robots.txt file in the root folder. Placing it elsewhere will not make it functional.

Why is Robots.txt file and Robots important to a webmaster?

Well, for a webmaster Robots.txt should be important because, it helps ensure better indexing of their websites, which means more information passed to search engines and thereby better search engine ranks for them.

It is possible for the webmaster to decide how their websites should be crawled, indexed and ranked by the search engines by the use of well-written Robots.txt files. So, it gives them complete (well almost) control over how a search engine "sees" their websites, which is very crucial.

How does a Robots.txt file look like?

This is a screenshot of two Robots.txt files. One (Screenshot A) has a minimal command usage, while the second (Screenshot B) has a deeper and complex command usage.



(Screenshot A)



(Screenshot B)

If you like to see more Robots.txt files, just type in the domain name followed by the /robots.txt filename in the browser of any website you like, and if they are using a Robots.txt file, then it would show up.(Ex: www.google.com/robots.txt, www.yahoo.com/robots.txt)

How and what can you control using Robots.txt?

You can control how search engines "crawl" your website by using the right commands on Robots.txt. For learning more about commands, let's first take a look at what the general syntax's are.

Robots.txt syntax and commands.

1. User-agent:

This command on the Robots.txt file specifies the general specification for the search engine robot.

For example:

User-agent: Google   (Means robots/crawlers from Google)

User-agent: Ask (Means robots/crawlers from Ask.com)

On the Robots.txt file, you can specify each user agent specifically, or invoke/address them generally by using the asterisk command.

User-agent: *  (Means all search engine robots/crawlers)

2. Allow/Disallow:

This command specifies the condition where it instructs a user agent to crawl/not crawl certain parts/all parts (as specified with the command) of the website. 

You can specify the directories within the website to be crawled/not crawled using the command.
For example:

User-agent: *

Disallow: /     (Means all the robots are not allowed to crawl everything that comes under the root folder, which is the entire website)

User-agent: *

Disallow: /temp/   (Means all the robots are not allowed to crawl the folder named "temp", while other parts are allowed to crawl)

How to set up a Robots.txt file?

Setting up a Robots.txt file can be tricky if you don't know the basic commands, so make sure you have studied the basics well before proceeding to set up a Robots.txt file.

Step 1

Open a new text document on your machine.

Step 2

In it, type these text, accurately.

User-agent: *

Disallow:

(This means that all user agents are allowed to crawl your entire website.)

Save it as "Robots.txt"

Step 3

Go to your server by accessing the file manager or the FTP, and go to the root folder. ( normally _public-html or http-docs or find out your's from your host.)


Step 4

Upload the "Robots.txt" file to the root folder.

Your Robots.txt file is now set up successfully. Note that we have given the command to allow allow all search engine robots to crawl the entire site without any restriction. If you would like to selectively disallow/block certain files/folders to be crawled, follow the commands below.

1. Exclude a file from an individual search engine.

User-agent: Google

Disallow: /thepathtoyourfile.html

Replace "Google" with your search engine preference and replace "thepathtoyourfile.html" with the actual path to your file. If you would like to block more than one file, you have to repeat this command (second line) with specific file names.

Ex: Disallow: /file1.html

Disallow: /file2.html

2. Exclude a section of your site from all spiders and bots

User-agent: *

Disallow: /1/2/dir-to-be-blocked/

Replace "dir-to-be-blocked" with the actual path to your directory that is to be blocked.

3. Allow all spiders to index everything

User-agent: *

Disallow:

OR

Leave the Robots.txt blank without any commands.

4. Allow no spiders to index any part of your site

User-agent: *

Disallow: /

This ensures that no spider would index anything at all on your site.

Free Robots.txt file generators

There are quite a number of free online Robots.txt file generators. Here is a list of few.

1. Mcanerin This tool lets you select search engine robots selectively that you'd like to block and create a Robots.txt file which you just need to copy paste.



2. Global promoter Robots.txt generator - Excellent tool that helps you generate a Robots.txt file with the help of a wizard.



Summary

Essentially, Robots.txt is an excellent tool to control how search engines scan your website, and gather up information from them. The more complex and careful you plan your website design, the better your search engine positions would be. But many websites simply ignore this and leave everything to the search engines to decide. Is that a good thing to do? I'd say it all depends on how you want it, if you think "not showing" a folders content to Google will avoid un necessary information being passed to it, and you know exactly how you can accomplish it, then why not use the options?

Related Articles
  What’s in Your Files? Putting Good Ideas to Use Can Help Your Creative Juice
  Backing Up Your Files Has Never Been Easier
  Egnyte Backup Service Review
  Why You Should Not Pay Your Bills At Work
  The Importance of Online Backup Services

Article Tags: internet consultant, robot files, robots, robotstxt



Related Forum Posts
Re: DropBox.com - Hey Evan, I just started using their service very recently. I'm currently using their free 2GB storage option. I mainly got it to send large-sized files to other people, as sometimes uploading to my web host server can become problematic and not work. I searched around for online storage options and landed on their site. Works great and very easy to share files with others. I currently have a 1TB external drive I backup all of my computer data on. Maybe someday I'll consider online backup, just for added protection of my files.
Re: Picking an external hard drive to backup my files - This is so funny that I am reading this, because as I type I am saving all files onto an external driver that I just purchased. I just got the WD (Westernet Digital), it holds 698gb worth of files. In my eyes thats a hell of a lot. They had I think 570gb for about $140 and the 698gb was $150 (They had a special on it), so obviously the more logical thing was to get the 698gb. So yeah I have WD, that I got from BestBuy. Would I buy one from eBay, I doubt it. For something like this I wanted to make sure it was brand new and noone had access to it.
Re: How do you back up your computer? - After deciding not to go with BackBlaze because they couldn't back up all of my files within the trial period (what's the point of a trial period of you can't actually try it out??) I have now given Dell a chance. I bought my new Dell laptop about 6 months ago and never initialized the Dell DataSafe Online application - until yesterday afternoon. My favorite thing about it so far is that before you back up it shows you how many files you have in different categories - ie documents, financial information, email files, videos, music, and photos. From there you can set your preferences as to what you want to back up. For me I have a large number of videos, photos, and music on my computer but I deselected them for the backup - To make the backups go more quickly I only chose the documents, financial information, and email files. This might be why my BackBlaze trial backup never completed. It works on the same premise that if you change a file it will recognize the change and only backup those changes. I have a free one year account that came with my Dell and so far I'm happy with it. You may want to consider looking into the program if you haven't found a suitable backup system yet!
Re: How do you back up your computer? - [quote="Evan":2o38iwps]Alright so my trial has expired and it didn't back up all my files during the trial period. Should I sign up to let the backup finish?[/quote:2o38iwps] Not to be difficult - but wouldn't the trial period be more effective if it actually included a complete backup to show how the product works? Another question - is there any way to delete your files if you discontinue the service?? Shri
Re: How do you back up your computer? - [quote="Evan":2rtu8bqc]After deciding not to go with BackBlaze because they couldn't back up all of my files within the trial period (what's the point of a trial period of you can't actually try it out??) I have now given Dell a chance. I bought my new Dell laptop about 6 months ago and never initialized the Dell DataSafe Online application - until yesterday afternoon. My favorite thing about it so far is that before you back up it shows you how many files you have in different categories - ie documents, financial information, email files, videos, music, and photos. From there you can set your preferences as to what you want to back up. For me I have a large number of videos, photos, and music on my computer but I deselected them for the backup - To make the backups go more quickly I only chose the documents, financial information, and email files. This might be why my BackBlaze trial backup never completed. It works on the same premise that if you change a file it will recognize the change and only backup those changes. I have a free one year account that came with my Dell and so far I'm happy with it. You may want to consider looking into the program if you haven't found a suitable backup system yet![/quote:2rtu8bqc] Hi Evan, So have you heard of any security problems with Dell's DataSafe Online Application though? And from your experience thus far, would you pay for subscription once your one year of free account expires?


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Remind Me...

Making the Most of Your Trade Show Experience

Are You Remarkable Enough?

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.