Tweet

Crawlability Issues - How to Make Sure That Search Engines Can Spider Your Site

Guest post by: Zeke Camusio

Article Overview: Just one simple mistake can seriously jeopardize your website’s crawlability. Follow these tips to make sure search engines can spider and index your site.

Crawlability Issues - How to Make Sure That Search Engines Can Spider Your Site

Search engine spiders aren't THAT smart. Spiders/bots are programmed to constantly crawl around search engines until they find content they can index. They aren't going to stop and wait around until you fix that broken link. That's not their job. It's YOUR job to make sure search engines can spider your site. In order for your site's pages to be crawled, they need to take visitors to different locations. Visitors should be able to easily navigate within your site or be taken to an external link/site. If you have bad or broken links, this seriously reduces your site's crawlability and your site could be permanently filtered out of SERPs (search engine result pages).

Crawlability errors are what kill your SEO - don't make the following mistakes or say good bye to being found in search engines.

• Pay attention to robots.txt files - That's the easiest way to mess up your crawlability. Robots.txt files direct search engine spiders how to view your site. One mistake can bump you out of search engines permanently so be extra careful when working with these files.

• Don't use too many variables in your URLs - Google can crawl long links but it's still not a fan of long URLs. It's better to use short URLs - they get clicked on more often in SERPs

• Don't add Session IDs in URLS - If you do use Session IDs in your URLs, store them in cookies. Spiders won't crawl URLs with Session IDs - they clog up the SERPs

• Watch out for code bloat - This can be a HUGE problem. Spiders are good at separating code versus content, but don't make it difficult for them either. Don't use so much code that your content isn't easy to find.

• Spiders can't follow Flash links or execute JavaScript - Designs that use Flash and JavaScript will stop spiders in their tracks. They also can't read text or images inside Flash. Stick with HTML links (links should preferably be text or images)

How to Fix Crawlability Mistakes

• Do NOT change your URLs unless they are broken. Keep in mind the following tips mentioned above. Keep them short, don't use too many variables, leave out Session IDs, etc.

• When creating URLs, use keywords and hyphens to separate keywords. Spiders recognize hyphens as spaces - don't use underscores, etc. Don't make your URLs look spammy with keywords.

• Fix your domain canonicalization. Are you going to use both versions of your domain? (don't use both!) Which domain gets the most inbound links, etc? That's the URL you should use as your primary domain. If you use a secondary domain, make sure you include a 301 redirect back to your primary domain.

• Use SEO Browser to check how search engines view your site

The Benefits of Sitemaps

Sitemaps are XML files that list URLS along with each URL's metadata. This allows search engines to crawl sites efficiently. Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.

If you want to be found easily by search engines, build an efficient navigation system. A logical and easy navigation structure will make it easier for spiders to crawl around your site.

Related Articles
  Create a Sitemap to Get your Site Indexed
  SEO Tip of the Day: 5 Steps for SEO Optimizing with DTD Statements
  Flash Sites and External Link Campaigns
  ••••••>For Best in Class SEO You Need to See Things They Way a Search Engine Spider Does
  SEO - What is SEO?

Article Tags: code bloat, crawlability, direct search, flash links, flash stick, good bye, google, html links, images, job, one mistake, pay attention, robots, search engine result, search engine spiders, search engines, seo, serps, session ids, variables



Related Forum Posts
Re: SES Toronto Next Week - In that case, how about the following? Track: Get Me Up to Speed * State of Search Marketing in Canada Track: Practical & Actionable * Beyond Linkbait: Getting Authoritative Online Mentions Track: Advanced * User Search Behavior * Social Media Success * Web 2.0 & Search Engines * Giving Credit Where It’s Due: Which Campaign Sold What?
Re: SEO Recommendation Needed - One who gives guarantee of top position in Search Engines surely does not know much about SEO
Re: Blogging for Entrepreneurs - Hi, Plus Search Engines LOVE blogs. If you can update your blog a few times a week that will go along way. Plus you like David said linking to internal pages (use good anchor text) can help other pages of your website be indexed faster & higher in the search engines. Jeff
Re: How To Pick Keywords To Rank For - For anyone that wants to learn SEO, Stomper Net is the true expert. I just found out over in the CYN forums that they are giving away their Stomping the Search Engines courses for just $1. I own a copy of these courses, and I can tell you they are rock-solid. I highly recommend them. This is way cheaper than paying a firm or "guru" to do it for you, and I'm a strong believer in learning stuff for yourself.
SES Toronto Day 2 - Well I’m off to Day 2 of Search Engine Strategies 2008. Here is what I have planned to attend today: 10:30am-11:45am - Search User Behavior How do searchers interact with search engines? New research is constantly revealing how searchers act. This session explores the latest studies and findings to provide tips and tactics for search marketers to consider. 12:45pm-2:00pm - SEM Toolkit: Marketers Share Competitive intelligence, keyword research, customer profiling, exotic analytics, and visual mouse tracking are just some of the tricks of the trade of search marketers. Guerrilla marketers on the panel show you the favorite tools in their kits. Many are low cost or free. 2:30pm-3:45pm - Web 2.0 & Search Engines An overused and amorphous term, “Web 2.0? nonetheless encompasses an evolutionary shift from the typical web user experience of a decade ago. Some aspects of the “new” web environment are great for improving your search traffic. Other developments may lead you to make serious SEO errors. Among other things, Web 2.0 is about new user interface designs that speed up user actions with techniques such as AJAX (Asynchronous Javascript and XML), which allow users to perform operations nimbly without loading a new HTML page. This session will cover AJAX, CSS, user-generated content, and other new trends in web design and user experiences that may require a re-think of your SEO strategy. 4:00pm-5:15pm - Site Clinic This interactive session takes volunteers from the audience and examines their websites live to provide general feedback about improving them to gain more traffic from search engines. As usual I’ll report my findings after the day is through!


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Tweet

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

An Intelligent Contact Sheet

ROSI Return on SUNK Investment

Five keys to business success

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.