Tweet

6 SEO Spider Tips on How A Search Engine Spider Works

Guest post by: John Alexander

Article Overview: Over the years there has been a lot of confusion about SEO and how search engines work. One of the things that comes up all the time is the subject of search engine spiders or as they are sometimes called search engine robots. Did you know that to a search engine robot, it does NOT see the world made up of Web sites or Web pages. Here are 6 Tips about Spiders.

6 SEO Spider Tips on How A Search Engine Spider Works

When a search engine robot visits your Web page, it is important to understand a few things if you are hoping to get your Web page indexed. Search Engine Robots are simply software programs that come out to your Web page, capture certain data and then return that data to the search engine from where
it came.
Do Search Engines "see" your entire Web site as a whole collection of pages or do you think they only see the whole world as a collection of individual pages?
Actually, the truth is neither "if you are thinking" based on how a search engine robot sees the
world. Think about this now.
A search engine robot does not see a world of pages or a world of Web sites.
A search engine sees its world made up only of URLs or (Uniform Resource Locators.)
This is one reason why search engine spiders cannot sometimes find their way out of what is called a Spider Trap. For example in some dynamic sites, you will often have variable strings at the end of a URL that are session identifiers and while the URL may be unique, the search engine does not realize it may be on exactly the same page over and over in a loop and may not be able to get out.
Do search engine spiders help determine what your rankings are? This is another wrong idea that some people have.
The search engine spider has one sole job and that is just to crawl content capture certain information and links and take it back to the main search engine.
How smart are search engine spiders?
The answer is they just are not very smart at all. They cannot think.
In fact most search engine spiders are updated very rarely with some powerful new feature. The reason why is because as most technology advances, the demands on a SE robot remain the same basically. They still only do exactly the same job they have always done. There are some spiders that are a little better than others at being more thorough and crawling more complexly.
But all things considered, why keep rebuilding something if what you have now works great?
Tips you should be aware of with spiders, crawlers and SEO and search engine optimization:
1. A spider is designed to crawl the Web. The smartest way to keep spiders consistently coming back to visit, is to focus on content freshness. Add new articles, pages, FAQs, good useful information to your Web site on a consistent basis - instead of doing major updates. Instead of adding 20 new articles all at once, try adding only 1 article a day (everyday) and watch what happens.
2. You do not need to include a robots revisit tag to your HTML. It does not work and the robot will come back more consistently with you adding fresh content, you may see it stay longer and actually dig deeper into your Web site.
3. Do you want to know one reason why Blogs are so busy and actively crawled by the spiders? It's because Blogs are often very fresh content sometimes being updated several times per day.
4. Remember that spiders and crawlers do NOT need to be submitted to for them to find you. There are actually good reasons not to use submission software but you are wiser to link to a new site from a Blog or another site that you know is already being indexed. Remember that you will still need to submit to directories though because there, you are dealing with human editors and not robots.
5. Make sure your Web site has a nice Site Map and then have at least one link to your site map on every page. Matt Cutts has said for example that this is one of the things that people most often forget. Remember, you have no idea whether a search engine spider will come in on your home page, or on your about us page or maybe even a policy page. But regardless of which way it comes in, it will be able to crawl ALL of your pages if you've created a good site map.
6. Do you think that it is best to get into Google immediately as quickly as you can?
Think about it for a minute. With Google, off page factors are VERY important.
Suppose Google's search engine robot visits you right away. But unfortunately, none of the other search engines have visited your page so even if you have some good links, nobody really knows it. Perhaps you need to think of a reverse approach and first submit to a few free or paid directories. Then you could begin getting your page indexed by some of the other spiders first, then finally when Google comes in it sees some nice off page factors and link reputation for which it will reward you for.

Related Articles
  ••••••>For Best in Class SEO You Need to See Things They Way a Search Engine Spider Does
  SEO Tip of the Day: 5 Steps for SEO Optimizing with DTD Statements
  Create a Sitemap to Get your Site Indexed
  ••••••>Free Top SEO Tip: Do Not Leave Abandoned Web Pages on Your Site; It’s Not Good for SEO
  Do You See What Everyone Else Sees On Your Website?

Article Tags: confusion, search engine robot, search engine robots, search engine spiders, web pages



Related Forum Posts
Re: Search Engine Tips and Ideas - Share Yours - Hi there, I answer this post, to start off. I am still learning a lot on all these subjects. Moreover, we have a wonderful tool to find out from previous posts. On the top right corner, use 'Advanced Search' and write 'Search Engine Tips'. It brings up all the previous posts with that information. Check it out. You will be surprised all the inherent information stored.
What Do You Want To Be Taught? - Hi, My name is Jeff Foster, I will be moderating this new section. I want to get your opinion on topics you would like to discuss & be taught on. I see a great need for small business owners to learn how to perform basic Search Engine Marketing (SEM) which would include the basics about Search Engine Optimization (SEO). What do you want to be taught on? Jeff
Re: Search Engine Optimization - If you are interested to get more information about Search Engine Optimization you can visit Search Engine Watch site gives you a great way to keep up with what’s going on in the search engine world. This site provides a ton of information about a very wide range of subjects related not only to search engine optimization, but also the flip side of the coin; subjects related to searching online.
Business Tips - How about: Tips for managers to handle employees more effectively? Tips on how to deal with difficult customers? Tips on how to deal more effectively with suppliers? The only three I have in mind right now, but will try to come up with something else. Chris
Re: Can someone explain Search engine optimization to me? - Search Engine Optimization (SEO) could be compared to a big popularity contest: A High School Computer Geek and a High School Football Captain who are trying to promote a party. The computer geek tells everyone in school & even hands out 100's of flyers about his party Friday night...what happens? his friends end up showing up and a few new people...even the new people leave right away because it is not what they are looking for. The High School Football Captain ends up just telling a few good buddies, and within hours the entire school not only knows about it, but everyone wants to come, and everyone does come & stay. One defenition of "optimize is "To make as perfect or effective as possibe." How effective are your article marketing campaigns? Could you be distributing content (that is what search engines index) more effectively? In a perfect world, who would you want promoting your website, web pages, and web content? Search Engine Optimization (SEO) is determining this perfect & effective way to do this. Jeff


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Intro to Search Engine Optimization

Halloween Howl Seven by Author Paige Agnew

Let's Skip the Offshore Horror Stories

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.