Does Your Website Have a Robots.txt File?

[WebBizIdeas.com]

Hi,

If you type in: www.YourDomain.com/robots.txt what appears? If nothing appears it means you do not have a robots.txt file installed; atleast correctly. If you do have a robots.txt file do you know what it means?

We just upload a tutorial on Robots.txt files you can read here:

http://www.webbizideas.com/building/robots.html

Let me know what you think.

[GT Bulmer]

Hello:

I appreciate the insight and links you have provided. I've been concerned about robots.txt for some time, but am too technically challenged and too busy to spend time researching it well enough to understand it well enough to properly understand what exactly I should be doing about it.

I'm sure the info at the link you provided will be beneficial to those who understand that sort of thing. I didn't get very far before realizing it's beyond my scope of understanding right now and with several deadlines pressing, I simply don't have time to spend trying to follow the technical line of thought to determine what I need to do and why I need to do it and how to do it right. Also, it's not something I will pay someone else to do before I understand it better.

Having said that, I think your tutorial is probably quite thorough and should be beneficial for those with some understanding and those willing to spend some study time on it. I will make note of it and come back to it at a later date.

Thank you for making it available to forum members.

GT

[topeyinka]

Hello:

I appreciate the insight and links you have provided. I've been concerned about robots.txt for some time, but am too technically challenged and too busy to spend time researching it well enough to understand it well enough to properly understand what exactly I should be doing about it.

I'm sure the info at the link you provided will be beneficial to those who understand that sort of thing. I didn't get very far before realizing it's beyond my scope of understanding right now and with several deadlines pressing, I simply don't have time to spend trying to follow the technical line of thought to determine what I need to do and why I need to do it and how to do it right. Also, it's not something I will pay someone else to do before I understand it better.

Having said that, I think your tutorial is probably quite thorough and should be beneficial for those with some understanding and those willing to spend some study time on it. I will make note of it and come back to it at a later date.

Thank you for making it available to forum members.

GT

This is quite informative. I have never heard about robots.txt before. Thanks for making this good information available for the forum.
I read everything about robots.txt on the website you gave it is amazing. I will inform my firends also.
Thanks a million times.

[ideasuniversity]

Whaow, This is amazing. I checked all my websites, non of them have any Robots.txt. Thanks for the information

[WebBizIdeas.com]

Hi GT,

If you think the material is too technical just go to the end where it talks about the software. There is a lot of software that will generate the code for you. Then it is as simple as using a FTP editor (ftpeditor.com) to upload them.

Jeff

[TannyL]

Thanks for the information.
I have this on my "maybe one day - to do list" for some quit time. I guess I am just being lazy

[GT Bulmer]

Thanks for the information.
I have this on my "maybe one day - to do list" for some quit time. I guess I am just being lazy

Hi, Tanny:

I think your main website might already have the robots.txt file set up.

A few days ago, when I first read this post by Jeff of WebBizIdeas I keyed in a few different websites to see what would happen.

My site and Carol's site default to our respective Plug-In Profit Site sales pages (that's not great, but I prefer that rather than going to a blank page or error page like it does with some sites.)

David's site goes to an error page (404, page not found).

But your's, Tanny, returns to your main page! And that is good. (I'm guessing your site must already have the robots.txt file set up properly).

And no, I KNOW you are not lazy - you are just stretched to your limits, as are many of us. hang in there!

GT

[GT Bulmer]

Hi GT,

If you think the material is too technical just go to the end where it talks about the software. There is a lot of software that will generate the code for you. Then it is as simple as using a FTP editor (ftpeditor.com) to upload them.

Jeff

Thanks for the extra info, Jeff. As Tanny just mentioned, I'll have to add it to my "to do" list because of how busy I am right now. But I have it bookmarked! Thanks again for posting the info and your links.

GT

[WebBizIdeas.com]

Hi,

GT, you brought up another good point...I think I will write a tutorial on this too:

404 Error Pages.

It is vital that your website (1) has a 404 error page and (2) I think it is important to have a custom one.

For example type in this:

http://www.webbizideas.com/tuyhjkld98we897w3kl

This is just a me punching in random letters & numbers on my keyboard. But the point is that no matter what you type in on webbizideas.com it takes you to a Custom 404 Error Page. This tells users (1) they typed in something wrong, accessing a page that is no longer there, an old bookmarked page, but since it looks like our site they may not get discouraged and leave.

What a 404 error page does is generate an html page when an error occurs on your website. This is vital when search engines crawl your website because if they come across an error and come to a normal blank error page that is negative points for you.

But if you have a normal html page (404 error page) it is ok; especially if you customize it. Even our page could be customized more. Adding a link to our contact page, faq page, or site map within our top section might help search engines bots & users continue to crawl our site.

Jeff

[Carol Aston]

Hi Jeff,

Thanks a lot for the advice about the robots.txt file. I've now uploaded one to my site using one of the free tools that you recommended Take a look when you get time please and let me know what you think.

I look forward to your tutorial on 404 Error Pages, it's about time I had one of those too.

Take care,

Carol

[WebBizIdeas.com]

HI Carol,

Your robot file is correct; no lets work on your Page Titles & Meta Data.

Do you have access to create different page titles for each page of your website? Do you have your FTP information...please do not post it here but just let me know if you have access.

Jeff

[Carol Aston]

Hi Jeff,

Yes I have access. Now you have me curious, why do you ask?

Take care,

Carol

[WebBizIdeas.com]

Hi Carol,

If you went to the super market and each isle was labeled "cereal" what type of products do you think would be in that Isle? If all the isles were labeled the same it would be confusing when determining where to find products, right?

The same is true with search engines. Search engines don't index websites they index web pages. Your home page contains a great Page Title but the problem is you have that title on each page

<title>Affiliate Program Business Opportunities | Work from Home Business</title>

You want to optimize each web page for a separate keyword or keyword phrase.

http://www.affiliatesthatserve.com/Articles.html

suggestion: Work at Home Business Articles | Run a Home Based Business
or
suggestion: Articles on Running Home Based Business | Work at Home

http://www.affiliatesthatserve.com/Programs.html

suggestion: Real Work at Home Programs | Online Business Opportunities
suggestion: The Best Work at Home Programs on The Internet

http://www.affiliatesthatserve.com/Tools.html

suggestion: Affiliate Program Marketing Tools | For Online Businesses

Do you know how to edit your Page Title? They are the text in between <title> </title>

Jeff

[Carol Aston]

Hi Jeff,

Thanks a lot for the advice, and your suggestions, this is very much appreciated and I'll make the changes you suggested. If you have any more advice I would love to hear it,

take care,

Carol

[GT Bulmer]

Awesome advice for Carol, Jeff, and very insightful for the rest of us, too! You presented it in a very simplified way that makes it easy to understand, and easy to envision how the changes are to be made ... and why they are necessary!

GT