Web Site Design by Rainbo Design

Preparing Your Website
For Search Engines

Start out right and your website will be a fast track to good rankings in the search engines. New webmasters are especially suseptible to myths, rumors and bad advice when it comes to search engine optimization. This SEO Tip contains my advice for all new webmasters, but especially for the webmaster who is new to search engine optimization and rankings. About 90% of what you need to know in order to do well in the search engines is contained on this first page. Just keep in mind that overnight successes in SEO are rare.


Preparing A Site for Search Engine Sumbission

Ideally, these are steps you should take before any user - whether human or search engine spider - ever sees your website. But, even if your site has been online for quite a long time already, it's never too late to start working on search engine optimization (SEO).

  1. Be sure your web site is ready for human visitors. The first thing I recommend is to add a robots.txt file to your site that blocks all search engine robots from your site until you are ready to start receiving traffic. Simply create a text file containing the following:

    UserAgent: *
    Disallow: /

    Then save this file as "robots.txt" and upload it to your site's root directory. That is, put it in the same directory on the server as the main page for your website. The search engines all recognize the robots.txt file for controlling how they access your site. This example of robots.txt prevents search engines from examining your site while it's still under construction. Just remember to remove this blocking instruction when you are ready to open your site by simply removing the "/" from the "Disallow:" line. If your site is already included in the search engines' index, skip this step.

  2. Insert a rel="canonical" tag in the <head> section of your main page to establish your preferred URL. As in:

    <link rel="canonical" href="http://wwww.yoursite.com/">

    This tag tells the major search engines that they should use the URL in the href attribute of this tag whenever they crawl and index this page, regardless of the URL they used to find it. You should select a preferred version of your URL - either with or without the "www." subdomain prefix, and use that whenever you link to your own site and whenever you ask another site to link to yours. Using this tag on your main page also eliminates the need to set up 301 redirects for variations of your main URL using index.html, index.php, etc. This tag is actually useful in many situations, but for now, just use it this way until you learn more about search engine optimization and canonicalization.

  3. Use Unique and Relevant Page Titles. The <title> tag is the most important piece of real estate on a web page. Not only does it carry a great deal of weight in the rankings, it is the first thing that users see when your pages appear in the search results. So you need to make sure that every page has a unique, keyword-rich title that will entice users to click on the link. Remember, rankings are not your ultimate goal. Getting users to actually visit your site is the real goal. Go ahead and include your site name in the title (at the end!), but the rest needs to be unique for every page. Get into the habit of making unique, informative, and enticing webpage titles and you'll have better results every time.

  4. Create a "description" <meta> tag in the <head> section of your web pages to create an appealing description of the contents of each page. The <meta> Description tag is not used by the search engines for ranking purposes, but they will often display the contents of this tag in the snippet of text that appears under your page <title> in the search results. Use this to your best advantage by creating a concise and compelling description of each page to attract clicks from search engine users. This is especially important for your main page, since it is the strongest page on your site and the most likely to appear in the search results. Be sure to include the primary target keywords for each page in this description, and keep it roughly between 100-200 characters in length; something more than a few words, but less than a paragraph. Do a few searches of your own to see which snippets are the most effective and you'll do fine.

  5. Your website's main page should have at least one paragraph of keyword-rich text that describes your site's topic or purpose to give the search engines a clear understanding of your site's content. The reason for doing this is that search engines can't read text rendered in graphics, such as images or graphical rollover menus and buttons. You also want to make sure that there is at least one plain link on the main page that points to the most important internal pages on your site.

  6. Make sure your HTML code is "search-engine-friendly". The World Wide Web is full of free HTML validators, which are programs that scan your website's pages for properly-written HTML code. So use them to your advantage. The best HTML and CSS validator is the W3C Markup Validation Service. You'll often prevent problems before they start just by letting one of these programs do their own special kind of spell-checking on the HTML code of your webpages. Browsers are designed to tolerate a high level of HTML errors, but search engine crawlers (a.k.a. "spiders" - the search engines' automated software that scans webpages) can be a bit fussier in how they decipher your HTML. The search engines don't give you extra credit for valid coding, but a stray < symbol or a missing quotation mark in your HTML code can cause a search engine spider to overlook or omit an entire section of your web page. Learn more about Search Engine Friendly Website Design.

  7. Once you have your site completed or nearly so, double-check that the links all work and the pictures all appear on the page when you access your web site over the Internet. Don't rely on the copy of the site on your own computer because some files may be on your computer, but not on the server. Test the actual online version of your website. If your website is too large to check out every link by hand, use one of the many automated link checkers like Xenu Link Sleuth which makes it simple to find broken links.

  8. Maintain A Local Back-Up Copy Of Your Website. Having a local back-up of your website's content is the best insurance you can have against several different problems that would be a disaster if you didn't have that data. Web servers do fail, and not every hosting service automatically backs up user data. Hackers are an ever-growing problem, and the only sure way to remove their malicious code is to restore your site from the known, clean back-up copies that you make yourself. Make it a habit to completely back up your entire site to your local computer - the HTML pages, graphics, video, mySQL databases, and anything else that you store on your site, and do it at least once a month. And be sure to back up your own computer as well.

  9. Set the robots.txt File To Allow the Search Engines to crawl (ie. "scan") your site. Now that your site is ready for the world to see it, you need to update the robots.txt file from Step 1 above to tell the search engines that they can crawl the pages from your site now. For most sites, the following robots.txt is best:

    UserAgent: *
    Disallow:

  10. Get Other Web Sites To Link To Yours. The World Wide Web's basic foundation is the hyperlink. Exchanging links with other web sites is a legitimate part of any early promotional program, so you need to prepare a page on your web site where you will put links to other web sites. Make sure there is a link on your web site's main page to this new links page. Once this page is installed you can contact the webmasters of other web sites and offer to exchange links. The best place to start is to visit other web sites whose topic is related to yours - even if they are direct competitors. These links will pay off in many ways, but links are vital to your site's performance in the major search engines. In fact, once you have another web site that links to yours, the major search engines will find your site on their own. You do not want to make link exchanges with every site on the Internet, so don't go overboard with them. Google's Webmaster Guidelines, in particular, prohibit "excessive" link exchanges. Focus on quality sites that are related to yours and limit them to a couple dozen or so, and you'll do well. See my Building Links to your Website SEO tips article for details.

  11. Submit your web site to online directories like The Open Directory Project A listing in high quality directories will help a bit with search engines, but obviously the listing itself can generate traffic, too. Glaciers are speed demons in comparison to The Open Directory Project's speed of processing of submissions, but the entry there can be valuable enough that it is worth the effort. However, never resubmit your site to The Open Directory Project. If you resubmit your site, you are essentially starting over at the end of the queue in waiting to be evaluated. So submit it once and forget it for at least a year.

    When choosing a directory, look for ones that review the quality of a website before they include them. Directories that accept any site in exchange for a link are not likely to do you any good. For example, consider quality directories like JoeAnt, Best of the Web, Jayde - The Business-to-Business Directory, and GoGuides. In highly competitive businesses, a listing in a paid directory like the Yahoo! Directory can be worthwhile because of the contextual setting of these links combined with the fact that Yahoo! does an editorial review of your site before your listing is accepted. Search engines are very good at analyzing the origins of links to determine the quality of those links for ranking purposes.

  12. Submit your website directly to the major search engines. Never pay for services that offer to submit your web site to thousands of search engines. Their use can actually impair your website's search engine ranking, and they often turn out to be a major source of SPAM in addition to being a complete waste of money. If you submit your site to Google, Yahoo!, and Bing, you've covered the sites that account for well over 90% of the search engine traffic on the Internet and none of them charge for submission to their search engine index. Search engines normally find new sites through links on other sites, so your submission may be superfluous, but it can sometimes get your site crawled sooner than it would otherwise. So take the 5 minutes to submit to Google, Yahoo!, and Bing.

    The best place to start is submitting your site to Google. If at least one other well-ranked web site links to yours, between your submission and that link, you should soon see a visit from the Googlebot and your main page will be included in the index within a week.

    Yahoo! Free Submission. Yahoo! also owns Alta-Vista and AllTheWeb, two modestly popular search engines, and also supplies search results to Hotbot.com. So if you get into Yahoo!, you're also in these other engines. Since Yahoo! is using Bing to supply their search engine on their main site, the status of these other engines is unclear, but Yahoo! is still crawling the web with their own robot, so it's worth submitting to them to insure you're seen everywhere possible.

    Submit MSN/Bing Search. Again, there's no charge for submitting your site to Microsoft's new search engine Bing, and since it's growing very quickly, you want to give them due attention.

    Inclusion in the search engines is not the same thing as ranking. While you have to be included in a search engine's index (database) in order to appear in the search results, inclusion does not guarantee good rankings. You need to take additional steps - the process of search engine optimization (SEO) - in order to compete against all of the other sites on the Internet. The other search engine optimization tips I present here will help you learn and understand that process.


Getting Your Site Indexed

The search engines store their information about the web in an array of databases called an "index". The process of being included in a search engine is called "indexing" When Google and the other search engines discover a link to your site on another well-ranked page that is already in their index, you might be indexed in as little as 24 hours. Once the Googlebot has passed through your website, you might be given a highly-positioned temporary ranking for your primary keywords. Google sometimes gives newly-discovered web sites a short period of enhanced visibility (listing plus high ranking) to see if actual human beings will want to see the web site based on the entry, rather than Google's ranking algorithms alone. It is not unusual for new websites to seem to disappear suddenly as Google periodically recompiles their master index. It doesn't mean you've been banned or penalized, it's just part of their process. New sites can often spend several months maturing before they really start to rank for their targeted keywords. The best course is to keep adding fresh, original content to your site, and to keep working on getting links from other well-ranked websites whose main topic is related to yours.

As I mentioned above, Google only pays attention to a few specific <META> tags. It will use the "description" <META> tag in the text that is displayed in the search results (a.k.a. the "snippet") if that description is relevant to the user's search. The Desciption <META> is also used to determine the uniqueness of a page - an important issue for Google in many ways - so don't simply copy and paste the same Description <META> on every page. Make each one unique, compelling, and relevant to the page content or don't have one at all. The Description <META> tag is not used for ranking purposes.

With rare exceptions, the search engines respect a "robots" <META> tag set to "noindex" or "nofollow". You do not need a robots <META> tag set to either "follow" or "index" because those are the default values. And the and "revisit" <META> tag is ignored by almost every search engine.

Finally, it's a very good idea to enroll in Google's Webmaster Tools. The Webmaster Tools console provides you with an enormous amount of information about the performance of your site in their search engine. You can see the keywords people use to find your site, which pages are linking to your site, and any problems Google is having with your site. They also provide you with several handy tools for enhancing your site. As your experience grows, you'll find this service to be increasingly valuable. Yahoo! and MSN/Bing offer similar services for webmasters that you might also find helpful. These are all free services that help you monitor your website's performance in the search engines.


This page was last updated on September 20, 2011

Need More Help?
You'll find more SEO Tips on the menu on the right side of this page.
You can also contact me with your SEO questions.

If you can't fix your website search engine problems on your own, my Search Engine Optimization Services can give your website what it needs to get your fair share of search engine traffic quickly, without disturbing your website's design, and without breaking your budget.

Call Richard L. Trethewey at Rainbo Design in Minneapolis today at 612-408-4057 from 9:00 AM to 5:00 PM Central time
to get started on your affordable website design package or search engine optimization program today!



 Comments or Questions?
 Contact Rainbo Design
  Share This Page!