Tweet

How Google Indexes Your Site

Written by: Paul Bliss

Article Overview: I can't tell you how many times I've answered this question in forums, so I figured since so many are asking, it would make for a great article.

How Google Indexes Your Site

First off, let's describe what we are talking about. A "bot" is a piece of software from a search engine that is built to go through every page of your site, categorize it, and place it into a database.

Google has three well known bots: The Adsense bot, the Freshbot and the DeepCrawl.

The Adsense bot, as you could probably guess, is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript within the Adsense code sends a message to the Adsense bot, and it will come within 15 minutes to index the page so that it can serve up the most relevant ads.

But, for this conversation we are only concerned about the DeepCrawl and the Freshbot.

The Freshbot crawls the most popular pages on your website. It doesn't matter if that is one page or thousands. Sites like Amazon.com and CNN.com have pages that are crawled every ten minutes, since Google has learned that those pages have that amount of frequent changes. A typical site should expect to have a freshbot visit every 1 to 14 days, depending on how popular those pages are.

What happens to your site on a Freshbot visit is that it finds all of the deeper links in your site. It places those links into a database so that when the DeepCrawl occurs, it has a reference.

Once a month, the DeepCrawl bot visits your site and goes over all the links found by the Freshbot. This is the reason why it can take up to a month for your entire site to be indexed in Google - even with the addition of a Google Sitemap.

So, be patient and keep on adding content to your site, and work on getting valuable in-bound links to your site - Google will reward you for it.

-To your online success!

Paul Bliss
www.SEOforGoogle.com

Related Articles
  Analyzing Your Competition – Check Indexes
  ••••••>Get Me On Top Free SEO Tip of the Day: Refrain from Duplication of Content When Performing SEO Techniques
  How Important are Sitemaps?
  What is web design and does it Matter?
  ••••••> #1 On Google and Great SEO Involves a Planned Website Evolution

Article Tags: Adsense, Google, site Google



Related Forum Posts
Most Useful SEO Tools For Webmasters - Hi, My name is Jeff, I work for WebBizIdeas.com and we are developing an article called "50 Most Useful SEO Tools for Webmasters." We have ranked & listed our top 30 SEO Tools but want to get your opinion on what SEO Tools are most useful to you; as Internet Business Owners. Google Webmaster Tools Google Analytics Google Alerts Yahoo Site Explorer SEOBook.com Keyword Research SEOBOOK.com SEO For Firefox Tool SEOBOOK.com Search Engine Rank Checker SEOBOOK.com Back Link Analyzer SEOChat.com Keyword Difficulty Compete.com Quantcast.com Alexa.com Archive.org Stompernet Site Seer Technorati.com PRWEB.com ADDTHIS.com SHARETHIS.com EzineArticles.com Scribd.com Squidoo.com YouTube.com StumbleUpon.com Reddit.com Digg.com Del.icio.us Wikipedia.org Flickr.com NewsVine.com Yahoo Answers NOTE: When you list your answers please include a description on what the site is, why it is important to you, and your experience please....we may include it in our article. Jeff
Re: SEO... - 1: Find the Best Keywords 2: Discover Your Competitors 3: Optimize Your Title 4: Optimize Your META Tags 5: Use Headings 6: Use Title and ALT Attributes 7: Nomenclatures 8: Create a Site Map Page 9: Include a robots.txt File 10: Install a sitemap.xml for Google MySQL/Oracle:
How Do You Check Google Ranking? - Now that Google is so responsive to your own search history, how can you get a good idea of what your Google ranking really is for your web pages? Whenever I check my ranking for specific keywords on Google, my web pages appear on the front page, or suspiciously high. My way around this is to check Google ranking via other computer Internet links, i.e., whenever I'm working in a library I will use a library computer to check. I check Google ranking before logging into my Gmail or other Google-related accounts. Does anybody else have any other tips for discovering your Google ranking? Nowadays, I check Bing.com for my webpage ranking as it gives a much less favourable (and so, I guess, "truer") ranking for my webpages than Google does...
Re: Cool Video - How Google Updates it's Search Algorithm - Great Video Evan! As per the recent Google Algorithm update. Google also give more value to content and even shows rich snippet in Google SERP's.
Re: From $0.30 to $300 / Day on Google AdSense Part 1 - How To M - HI Evan, Look forward to hearing more. I like the point that you brought out with "testing." Google has some amazing tools that can help website owners track everything. From Google Adsense reporting to Google Analytic to Google Conversion and most recently Google Goal Tracking. Between all of those tools a website owner should know exactly what type of traffic is converting & making them money and what type of traffic is not. I even use Google PPC to estimate organic search engine traffic! Website owners would do well to follow your lead and create a base, and then test off that base in regards to advertising. Jeff


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Live To Work Or Work To Live?

WHAT IS MANAGEMENT CONSULTING

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.