How Search Engines Index Websites

What is a Search Engine?

The term "search engine" is often used interchangeably to describe crawler-based search engines and human-edited web directories. However, these two search systems use very different ways to gather their listings.

Human-edited directories such as Dmoz.org rely on submissions from humans to populate the directory's categories. Crawler-based search engines such as Google use special software called "bots" or "spiders" to "crawl" the Internet, to look for webpages to add to their listings.

Directories contain information about websites, whereas crawler-based search engines gather information from webpages. They don't necessarily grab all the information on each webpage, but they take a significant amount and apply complex algorithms to index the information.

Parts of a Search Engine All search engines consist of three parts: a database of webpages, a spider operating on that database, and a series of search engine software that decide how search results are displayed.

First, a spider visits a webpage, reads it and then follows the links to other pages within the website. All the data that the spider has gathered is stored in the database, which contains a copy of every webpage that the spider has found. The spider will often return regularly to a website to look for any changes and update the database accordingly.

The search engine software sifts through the millions of webpages stored in the database, identifying the body text, links and other content on the page. It does this so it can find matches to a search and ranks them in order of their relevancy. Page titles, meta-descriptions and other various elements play a part in determining where a website should be ranked.

One advantage of having spiders revisit your site is that you can make changes to your webpage in order to appear higher in their relevancy rankings, then see if your changes worked. This is what search engine optimization is all about.

Although every search engine is made up of these three parts, there are differences and biases in how webpages are evaluated. This explains why the same search phrase will yield different results on different search engines. Google uses over 150 criteria to evaluate a webpage.

Some pages however are excluded from the database either by policy or because the spiders cannot access them, such as Flash pages or webpages with URLs containing special characters like question marks or ampersands (&). These factors prevent a website from being viewed as search-engine friendly.

The Importance of Linking If a webpage is never linked to from any other webpage, the spiders will never find it. The only ways a brand new webpage can get indexed by the search engines is if it is linked from within the website or the URL is sent to the search engine companies as a request to be included in their index.

It's important to know that the links that count in terms of search rankings, are the ones pointing to your website.

Online Resources about Search Engines Articles about search engines abound on the Internet. However, one of the more interesting ones is a Search Engines Tutorial (www.lib.berkeley.edu at the UC Berkeley Online Library.

This tutorial looks at the features of all the major search engines (Google, Yahoo, Ask.com) and has a document summarizing what makes a good search engine.

Danny Sullivan, former editor of SearchEngineWatch.com, wrote a series of detailed articles explaining how search engines work. A little outdated but still relevant, he also explains how to make your website pages more search-engine friendly.



How Search Engines Index Websites - To learn more about this author, visit Pasquale Spadafora's Website.

Like this article? Share it with your friends

E-Mail | Print | More

Related Articles
Search Engine Optimization Overview
	These days your website must not only look good, it has to be visible to the search engines. Search Engine Optimization (SEO) is a means of ranking your website in the top results of the major search engines for cer...
Carefully Choose to Whom You Link To
	Websites that you link to create your neighborhood of websites if you will. If websites that you link to are known to be spammy websites by the search engines then the search engines may also think that your web sit...
Search Engine Friendly Pages
	Major search engines use programs called crawlers or robots to index websites to list on their search result pages. They follow links to a page, reads the content of the page and record it in their own database, pul...
How Fresh Content May Catapult Your Website Ranking At The Top!
	It used to be that the content didn't really matter and this is why unethical black hat strategies were so effective. Nowadays though content does matter and search engines are always going to prefer original conten...
5 Costly Search Engine Optimization Mistakes That May Doom Your Business
	As a website owners you already know the importance of traffic. Getting a steady flow of visitors to your site is the only thing that really matters. If you can't generate traffic your online business is destined to...

---

Related Forum Posts
	Re: SES Toronto Next Week
	Re: Best Paid Directory Listings For Keyword Rankings
	Re: Business Directories
	SES Toronto Next Week
	Re: Kevin's Case Study #9 - How do you increase forum traffic?
	Re: Kevin's Case Study #9 - How do you increase forum traffic?
	SES Toronto Day 2
	Amazed!