Tweet

8 Steps to Getting Your Web Site Crawled

Written by: Karen Scharf

Article Overview: In order to analyze the content on your web site and figure out which pages to display (and in which order) for certain search terms, the search engines (SEs) use a program called a web crawler.

8 Steps to Getting Your Web Site Crawled

In order to analyze the content on your web site and figure out which pages to display (and in which order) for certain search terms, the search engines (SEs) use a program called a web crawler. You might have also heard these referred to as spiders or web-bots.

Every SE has its own web crawler program and its own formula for ranking pages. However, while page factors might be ranked differently by different engines, the idea behind the web crawlers is basically the same. Getting your site crawled often is a good indicator that the SE finds your site important. And since important sites get better rankings, getting your site crawled frequently is definitely a good thing.

While there is no way you can make the SE crawl your site, there are a few things you can to do encourage the spiders to visit your site more often:

First and foremost, keep your content up to date. Add unique content often and on a regular basis. And be sure to ping Google each time you update your site.

Get as many backlinks as possible from other relevant sites that are crawled regularly. There are several tools available that will track and measure the crawl rate of other websites.

Use a unique title tag for each page. Also, while not as important as the title, each page should also have its own keywords and description tag. This will help the spiders "know" that each page is a unique and individual entity.

Keep your pages as small as possible. The spider will not spend an unlimited amount of time at your web site. If it's held up by your huge images, monstrous PDFs or even excessive text, it will simply abandon the page and move on.

Pay close attention to your internal links and avoid any duplicate content issues. If you have multiple pages with the same content (often used for testing and/or tracking purposes) be sure to use your robots text file to keep the SEs away.

Create a sitemap for the content you want crawled and make sure your content is compliant.

Monitor your web server. Obviously, your server needs to be up and running when the spiders come a-crawling. There are many commercial providers who will monitor your server and notify you when there is a problem. In addition, ensure that your server handles all of your error pages correctly. Refer to the Google Webmaster Tools report of unreached pages for an idea of what errors are being returned.

And while we're discussing Google Webmaster Tools, don't forget to use it to monitor your crawl rate, adjust it if necessary, test and track and see what works best for your site.

Time to implement: As with all SE optimization, working toward an optimal crawl rate is not a one-time job. You will need to add content approximately three times a week and update your site map as needed. Monitoring your web server is an ongoing task.

Related Articles
  Understanding Sitemaps and How They Help Your Website
  Create a Sitemap to Get your Site Indexed
  How Google Indexes Your Site
  The Importance of Fresh Content
  Site Map Tips

Article Tags: internet marketing, website crawled



Related Forum Posts
Re: Hello from the UK - Hello! What part of the UK are you from? Site looks good, how are you marketing it and what do what want to achieve?
Re: Drop out Billionaire - I reveived Bill Bartmann's DVD "9 Steps To Achieve Any Goal" as part of my monthly pack from Success University a few months back. It is a great presentation. I've watched it several times but not for a while so I'll view it again in the coming week. He is one of the greats. Like the best success mentors, he speaks with authority because he has come through much tribulation. His program sounds like a great bargain at $100 a year.
New Guy Here - Automobile Industry - Hi Everyone, My name is Ray and I work as a sales trainer in the automobile industry. I have just completed a new training curriculum for automobile salespeople in the industry. It is called The Baby Steps of Selling Automobiles. I am beginning my marketing efforts through press releases and seminar formats. I joined the community due to my continued desire to work independently creating my own happiness and growth. Thank you for having me and I look forward to more correspondence in the future.
Re: Eng Lit Academic Site Up For Review - Hi David Site looks good but I agree with Zacman, the header definitely needs something more on it. It doesn't need to be beautiful but it does need to catch the eye to attract people to read further. MichelleJ
Re: Alexa Ranking Goals - My Alexa ranking goal Site: TJHarrisMarketing (dot) com Member:Timothy Harris Goal: 100,000 Currently: 670,695 and im also currently working on other wordpress websites


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Life, Conflict and Work

Working Across Borders

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.