Feedback Form
Home Features Mastermind Forums About Advertise Blog Network Contact Be An Author

6 SEO Spider Tips on How A Search Engine Spider Works

6 SEO Spider Tips on How A Search Engine Spider Works

When a search engine robot visits your Web page, it is important to understand a few things if you are hoping to get your Web page indexed. Search Engine Robots are simply software programs that come out to your Web page, capture certain data and then return that data to the search engine from where
it came.

Do Search Engines "see" your entire Web site as a whole collection of pages or do you think they only see the whole world as a collection of individual pages?

Actually, the truth is neither "if you are thinking" based on how a search engine robot sees the
world. Think about this now.

A search engine robot does not see a world of pages or a world of Web sites.

A search engine sees its world made up only of URLs or (Uniform Resource Locators.)

This is one reason why search engine spiders cannot sometimes find their way out of what is called a Spider Trap. For example in some dynamic sites, you will often have variable strings at the end of a URL that are session identifiers and while the URL may be unique, the search engine does not realize it may be on exactly the same page over and over in a loop and may not be able to get out.
Do search engine spiders help determine what your rankings are? This is another wrong idea that some people have.

The search engine spider has one sole job and that is just to crawl content capture certain information and links and take it back to the main search engine.

How smart are search engine spiders?
The answer is they just are not very smart at all. They cannot think.

In fact most search engine spiders are updated very rarely with some powerful new feature. The reason why is because as most technology advances, the demands on a SE robot remain the same basically. They still only do exactly the same job they have always done. There are some spiders that are a little better than others at being more thorough and crawling more complexly.

But all things considered, why keep rebuilding something if what you have now works great?

Tips you should be aware of with spiders, crawlers and SEO and search engine optimization:

1. A spider is designed to crawl the Web. The smartest way to keep spiders consistently coming back to visit, is to focus on content freshness. Add new articles, pages, FAQs, good useful information to your Web site on a consistent basis - instead of doing major updates. Instead of adding 20 new articles all at once, try adding only 1 article a day (everyday) and watch what happens.

2. You do not need to include a robots revisit tag to your HTML. It does not work and the robot will come back more consistently with you adding fresh content, you may see it stay longer and actually dig deeper into your Web site.

3. Do you want to know one reason why Blogs are so busy and actively crawled by the spiders? It's because Blogs are often very fresh content sometimes being updated several times per day.

4. Remember that spiders and crawlers do NOT need to be submitted to for them to find you. There are actually good reasons not to use submission software but you are wiser to link to a new site from a Blog or another site that you know is already being indexed. Remember that you will still need to submit to directories though because there, you are dealing with human editors and not robots.

5. Make sure your Web site has a nice Site Map and then have at least one link to your site map on every page. Matt Cutts has said for example that this is one of the things that people most often forget. Remember, you have no idea whether a search engine spider will come in on your home page, or on your about us page or maybe even a policy page. But regardless of which way it comes in, it will be able to crawl ALL of your pages if you've created a good site map.

6. Do you think that it is best to get into Google immediately as quickly as you can?

Think about it for a minute. With Google, off page factors are VERY important.

Suppose Google's search engine robot visits you right away. But unfortunately, none of the other search engines have visited your page so even if you have some good links, nobody really knows it. Perhaps you need to think of a reverse approach and first submit to a few free or paid directories. Then you could begin getting your page indexed by some of the other spiders first, then finally when Google comes in it sees some nice off page factors and link reputation for which it will reward you for.





6 SEO Spider Tips on How A Search Engine Spider Works - To learn more about this author, visit John Alexander's Website.

Like this article? Share it with your friends

Article Feedback
 Article Feedback No article feedback found.
  Leave Your Feedback
article feedback

Article Feedback
John Alexander
John has taught keyword research and SEO skills to small groups of business owners and Webmasters from over 80 different countries world wide since 2002. John is also the Director of Search Engine Academy ; Co-director of Training at Search Engine Workshops offering live, SEO Workshops with his partner SEO educator Robin Nobles, author of the very first comprehensive online search engine marketing courses at SEO Training Online and the SEO Workshop Resource Center.
I look forward to hearing from you! - Visit John Alexander's Website

George Ludwig
George Ludwig is a recognized authority on sales strategy and peak performance psychology. An international speaker, trainer, and corporate consultant, he helps clients like Johnson & Johnson, Abbott Laboratories, Northwestern Mutual, CIGNA, and numerous others improve sales force effectiveness and performance. Though it's George's strategies and processes that help corporations increase productivity and performance, it's his tremendous energy and dynamism that spark the transformation. Again and again, clients remark on his amazing ability to unleash human capacity and inspire men and women to break out of their comfort zones. The result is a whole new type of salesperson. His customized presentations teach achievers to make stunning advances in their lives. From helping salespeople realize cherished dreams to helping corporations exponentially accelerate revenue streams, George Ludwig leaves audiences and individuals empowered, emboldened, and clamoring for more. George is the best-selling author of Power Selling: Seven Strategies for Cracking the Sales Code and Wise Moves: 60 Quick Tips to Improve Your Position in Life & Business. - Visit George Ludwig's Website

Stephanie Robey
Stephanie Robey is President and CoFounder of Pivot Positive, LLC - an Internet marketing business focused on helping people start work at home ventures. Previously, she was employed at The Search Agency with over 20 years experience in graphic design and 10 years experience in online marketing. She was responsible for launching the Conversion Path Optimization (CPO) unit where she and her team have conducted hundreds of optimization tests for online companies across multiple verticals.

She is a successful entrepreneur having started and sold 2 companies and remains on the board of directors of the third, PhotoSpin.com   Stephanie began her career in the direct marketing realm creating and producing direct mail for many of the major cable television companies and directly attributes her understanding of Internet marketing to those early offline experiences.  Stephanie is a graduate of San Diego State University with a BFA in Graphic Arts and also holds an Executive MBA from the Graziadio School of Business and Management at Pepperdine University.

Read Steph's Blog
Meet Steph and Dave
Sign up for our Free 7-Day BootCamp: Self Employed & Rich
- Visit Stephanie Robey's Website


To learn more about the Evan Elite Author Program please contact us.

About The Author


John Alexander
(Visit John's Website) John has taught keyword research and SEO skills to small groups of business owners and Webmasters from over 80 different countries world wide since 2002. John is also the Director of Search Engine Academy ; Co-director of Training at Search Engine Workshops offering live, SEO Workshops with his partner SEO educator Robin Nobles, author of the very first comprehensive online search engine marketing courses at SEO Training Online and the SEO Workshop Resource Center.
I look forward to hearing from you!

John Alexander is a Platinum author on EvanCarmichael.com
About The Author

View Author Blog
KEI Observation Deck - Friendly High-Altitude KEI Observations for Keyword Researchers Everywhere
KEI Observation Deck
View Author Blog

View Author Video
John Alexander Video - In this Video, John Alexander explains how Wordtracker can also be used to find related synonyms (based on what a search engine like Google understands through AI or Artificial Intelligence. Are you looking for synonyms based on human intelligence or a Search Engine's Artificial Intelligence. It makes a difference.
View Author Video

Free Downloads


John Alexander's

Complete
List Of
SEO
Articles

Name
Email
If you enjoyed this article, get John Alexander's Complete List of SEO Articles For FREE!

More John Alexander
Interesting new KEI Metrics and Keyword Research upgrades at Wordtracker
How to position your Web site for a Media Blitz Part 1
Fine Tune your Blog and Build More Readership using Contests
10 Tips help build the foundation for a new SEO career
SEO Certification Has your SEO Team been properly trained
10 Actions You Can Take to Tip Your Scales For Online Success Starting This Year
What is your Philosophy on Search Engine Marketing
How to Improve SEO Results for Insurance Related Web Sites
SEO Technique What is the Smartest Way to Get Started
Separating 7 SEO Truths from 7 Common Misconceptions
Free Downloads


 
 
 


Evan Elite Authors
Jeff Foster  
John Alexander  
John Brennan  
Evan Elite Authors

Become An Author
Have you written articles that would be of value to entrepreneurs? Become an expert on our site by publishing them! Expose yourself to a wide audience, drive more traffic to your website and get more sales! Click Here for details.
Become An Author

Evan's Latest Video
Modeling the Masters: Learn the true secrets behind Walt Disney's business success factors & grow your company! Video produced by Phanta Media
Evan's Latest Video

Business Opportunities
"Learn straight from Evan how you can Make a Full Time Income (And More) from a Website"

How to Start An Online Business

Click Here To Learn More
Business Opportunities



Evan's Newsletter
Get advice & tips from famous business owners, new articles by entrepreneur experts, my latest website updates, & special sneak peaks at what's to come!
Name:
Email:
Evan`s Newsletter

Free Downloads
Host Beneficiary Icon Host Beneficiary
SWOT Analysis Template Icon SWOT Analysis Template
Put Yourself Into the Top 3% Icon Put Yourself Into the Top 3%
Reference Check Form Icon Reference Check Form
Dismiss a Franchise Icon Dismiss a Franchise
Free Downloads - Complete List

Entrepreneur Tools and Guides
Top 50 Franchising Blogs
Top 50 Franchising Blogs
Top 50 Franchising Blogs
 
Write The Press Release
Write The PR
Press Release Builder
 
Entrepreneur Tools and Guides

SEO For Africa
SEO For Africa
Tausi Nabbanja's Group Jinja, Uganda,
SEO For Africa

If I Were A Startup...
Kerry Shapansky, $2.0 to $51 Mil in 5 years
Kerry Shapansky
$2.0 to $51 Mil in 5 years
Razor Suleman, $143k to $5.4 Mil in 5 years
Razor Suleman
$143k to $5.4 Mil in 5 years
If I Were A Startup... - Complete List

Famous Entrepreneurs
Chris Gardner, Gardner Rich
Chris Gardner
Gardner Rich
George Lucas, Lucasfilm
George Lucas
Lucasfilm
Famous Entrepreneurs - Complete List

Entrepreneur Advice
Paul Kedrosky, Venture Capitalist
Paul Kedrosky
Venture Capitalist
Donald J. Trump, Trump University
Donald J. Trump
Trump University
Entrepreneur Advice - Complete List

Popular Articles
(Premium Authors)

     To Be (Customer-Focused) - Or Not To Be... What a Question
By Ray Miller
     Why Customer Focus Differentiates
By Ray Miller
     Become a Customer-Focused Leader
By Ray Miller

Have A Suggestion?
Toronto Salsa Classes / Toronto Salsa Lessons Email us your ideas on how to make our website more valuable! Thank you Sharon from Toronto Salsa Lessons / Classes for your suggestions to make the newsletter look like the website and profile younger entrepreneurs like Jennifer Lopez and Sean Combs!
Have A Suggestion?

More Evan Carmichael
More Information