---

Guest post by: Karen Scharf Become an Author

Tweet

---

---

In order to analyze the content on your web site and figure out which pages to display (and in which order) for certain search terms, the search engines (SEs) use a program called a web crawler. You might have also heard these referred to as spiders or web-bots.

Every SE has its own web crawler program and its own formula for ranking pages. However, while page factors might be ranked differently by different engines, the idea behind the web crawlers is basically the same. Getting your site crawled often is a good indicator that the SE finds your site important. And since important sites get better rankings, getting your site crawled frequently is definitely a good thing.

While there is no way you can make the SE crawl your site, there are a few things you can to do encourage the spiders to visit your site more often:

First and foremost, keep your content up to date. Add unique content often and on a regular basis. And be sure to ping Google each time you update your site.

Get as many backlinks as possible from other relevant sites that are crawled regularly. There are several tools available that will track and measure the crawl rate of other websites.

Use a unique title tag for each page. Also, while not as important as the title, each page should also have its own keywords and description tag. This will help the spiders "know" that each page is a unique and individual entity.

Keep your pages as small as possible. The spider will not spend an unlimited amount of time at your web site. If it's held up by your huge images, monstrous PDFs or even excessive text, it will simply abandon the page and move on.

Pay close attention to your internal links and avoid any duplicate content issues. If you have multiple pages with the same content (often used for testing and/or tracking purposes) be sure to use your robots text file to keep the SEs away.

Create a sitemap for the content you want crawled and make sure your content is compliant.

Monitor your web server. Obviously, your server needs to be up and running when the spiders come a-crawling. There are many commercial providers who will monitor your server and notify you when there is a problem. In addition, ensure that your server handles all of your error pages correctly. Refer to the Google Webmaster Tools report of unreached pages for an idea of what errors are being returned.

And while we're discussing Google Webmaster Tools, don't forget to use it to monitor your crawl rate, adjust it if necessary, test and track and see what works best for your site.

Time to implement: As with all SE optimization, working toward an optimal crawl rate is not a one-time job. You will need to add content approximately three times a week and update your site map as needed. Monitoring your web server is an ongoing task.

Related Articles

  Understanding Sitemaps and How They Help Your Website
  Create a Sitemap to Get your Site Indexed
  How Google Indexes Your Site
  The Importance of Fresh Content
  Site Map Tips
  How a Sitemap Differs From a sitemap
  Increasing Traffic Generation – Grow Your Online Business
  Search Engine Optimization (SEO) Basics – Part 2
  Site Speed
  Yahoo gets Binged – how will this affect your SEO
  Six Quick Ways Your Website Benefits From Blogging
  Start your SEO from Sitemap submission by Free SEO Software Tool
  Web Design Tips And Strategies For Professional Websites
  5 Reasons Why Your Site is not Ranked in Google
  Secrets to Filling your Coaching Practice
  Internet Marketing. My Own Experience.
  Top 5 White Hat Techniques in SEO
  Search Engine Optimization for Google, Yahoo and Msn
  Basics of Search Engine Optimisation (SEO)
  Search Engine Friendly Web Design

Related Forum Posts

Re: Hello from the UK
Re: Drop out Billionaire
New Guy Here - Automobile Industry
Re: Eng Lit Academic Site Up For Review
Re: Alexa Ranking Goals

Share this article. Fund someone's dream.

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Why Prospects Don’t Tell You the Truth

My Favorite Motivational Quotes

How should CEOs select business metrics?

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.