Tweet

How Search Engines Index Websites

Written by: Pasquale Spadafora

Article Overview: For your online business to be successful it needs be optimized and updated on a regular basis to ensure it maintains a good position on the major search engines. Positioning your website competitively, through SEO and other methods of online marketing, starts by understanding what a search engine is and how one works.

How Search Engines Index Websites

What is a Search Engine?

The term "search engine" is often used interchangeably to describe crawler-based search engines and human-edited web directories. However, these two search systems use very different ways to gather their listings.

Human-edited directories such as Dmoz.org rely on submissions from humans to populate the directory's categories. Crawler-based search engines such as Google use special software called "bots" or "spiders" to "crawl" the Internet, to look for webpages to add to their listings.

Directories contain information about websites, whereas crawler-based search engines gather information from webpages. They don't necessarily grab all the information on each webpage, but they take a significant amount and apply complex algorithms to index the information.


Parts of a Search Engine

All search engines consist of three parts: a database of webpages, a spider operating on that database, and a series of search engine software that decide how search results are displayed.

First, a spider visits a webpage, reads it and then follows the links to other pages within the website. All the data that the spider has gathered is stored in the database, which contains a copy of every webpage that the spider has found. The spider will often return regularly to a website to look for any changes and update the database accordingly.

The search engine software sifts through the millions of webpages stored in the database, identifying the body text, links and other content on the page. It does this so it can find matches to a search and ranks them in order of their relevancy. Page titles, meta-descriptions and other various elements play a part in determining where a website should be ranked.

One advantage of having spiders revisit your site is that you can make changes to your webpage in order to appear higher in their relevancy rankings, then see if your changes worked. This is what search engine optimization is all about.

Although every search engine is made up of these three parts, there are differences and biases in how webpages are evaluated. This explains why the same search phrase will yield different results on different search engines. Google uses over 150 criteria to evaluate a webpage.

Some pages however are excluded from the database either by policy or because the spiders cannot access them, such as Flash pages or webpages with URLs containing special characters like question marks or ampersands (&). These factors prevent a website from being viewed as search-engine friendly.


The Importance of Linking

If a webpage is never linked to from any other webpage, the spiders will never find it. The only ways a brand new webpage can get indexed by the search engines is if it is linked from within the website or the URL is sent to the search engine companies as a request to be included in their index.

It's important to know that the links that count in terms of search rankings, are the ones pointing to your website.


Online Resources about Search Engines

Articles about search engines abound on the Internet. However, one of the more interesting ones is a Search Engines Tutorial (www.lib.berkeley.edu/TeachingLib/Guides/Internet/SearchEngines.html) at the UC Berkeley Online Library.

This tutorial looks at the features of all the major search engines (Google, Yahoo, Ask.com) and has a document summarizing what makes a good search engine.

Danny Sullivan, former editor of SearchEngineWatch.com, wrote a series of detailed articles explaining how search engines work. A little outdated but still relevant, he also explains how to make your website pages more search-engine friendly.

Related Articles
  SEO Tip of the Day: 5 Steps for SEO Optimizing with DTD Statements
  Search Engine Optimization Overview
  Search Engine Friendly Pages
  SEO – On-site and Off-site Optimization
  SEO - Using Meta Tags

Article Tags: algorithms, different ways, elements, google, meta descriptions, page titles, relevancy rankings, search engine optimization, search engine software, search engines, search systems, software sifts, special software, spiders, submissions, term search engine, web directories



Related Forum Posts
Re: SES Toronto Next Week - In that case, how about the following? Track: Get Me Up to Speed * State of Search Marketing in Canada Track: Practical & Actionable * Beyond Linkbait: Getting Authoritative Online Mentions Track: Advanced * User Search Behavior * Social Media Success * Web 2.0 & Search Engines * Giving Credit Where It’s Due: Which Campaign Sold What?
Re: SEO Recommendation Needed - One who gives guarantee of top position in Search Engines surely does not know much about SEO
Re: Blogging for Entrepreneurs - Hi, Plus Search Engines LOVE blogs. If you can update your blog a few times a week that will go along way. Plus you like David said linking to internal pages (use good anchor text) can help other pages of your website be indexed faster & higher in the search engines. Jeff
Re: How To Pick Keywords To Rank For - For anyone that wants to learn SEO, Stomper Net is the true expert. I just found out over in the CYN forums that they are giving away their Stomping the Search Engines courses for just $1. I own a copy of these courses, and I can tell you they are rock-solid. I highly recommend them. This is way cheaper than paying a firm or "guru" to do it for you, and I'm a strong believer in learning stuff for yourself.
Re: Using Twitter To Double Your Traffic? - I'm sure in many cases Twitter sends more traffic than Search Engines because most people don't have good enough SE rankings to get SE traffic. But anyone can get Twitter traffic. The problem is that Twitter traffic won't convert for the most part and SE traffic will. I'd venture to say that 98% of the time getting just 100 visitors from search is better than getting 2000+ from Twitter.


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

How do you keep it together on a daily basis?

You Have A Website What Now

Emotional Energy is Our Engine

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.