Finding Keywords for Search Engine Marketing

There are always two sides to any relationship, and that certainly holds true for the relationship between your website and search engine users. In the first place, you have a theme for your website that includes certain topics, product lines, or services. These themes comprise your site's primary keywords and they are certainly where you begin when you design your webpages. And since you, as the webmaster/author, are bound to know your topic very well, you will naturally tend to use the technically correct terminology, the most precise phraseology, the premium buzzwords, the jazzy jargon to use in making your website the cin qua non in your field of endeavor. But, on the other hand, there are the terms which real live users, lacking your well-earned expertise and experience, use when they speak, and these are the terms they will enter into search engines. Thus, these are keywords and phrases to which you must also incorporate into your search engine optimization strategies. Obviously, the trick is in finding out precisely which terms John Q. Public is using in search engines to find websites like yours, and which of them are used often enough to be worth including in your keyword targeting. And just where, you might well ask, does one obtain such information? Well, an excellent place to look is the server log file for your website.

The server log for your website will include a specific piece of information for each webpage request it has received that tells you where on the Internet the request originated called the "REFERER" (sic). In some cases, the requests will originate from other pages on your website. Other times, users will simply type the webpage address (URL) into their browser or use a stored bookmark. But in terms of researching keywords, the most important case is when a webpage request comes from another website - specifically when it comes from a Search Engine Result Page (SERP). These requests will contain the keywords that the user entered and caused the search engine to display a link to a page on your website. As your website becomes bigger and stronger and more well-established in the search engines, you will probably discover that many users are using search terms that you hadn't really considered to be important.

The server log file is going to be very big. Depending on how busy your website is and how often your host rotates the log, it will probably be several megabytes. So I usually recomment that you use a full-strength word processing program like Microsoft Word to examine a log file yourself, because its built to deal with very large files. A much better solution is to use a Server Log Analysis program like Webalyzer or AWStats which is designed to analyze your logs quickly and easily, with special functions to decipher search engine activity.

Each entry in the server log file represents a request for a file on your website. The entry will tell you the IP address of the requester, the time and date of the request, the name of the file being requested, and the origin of the request - technically known as the "referrer" (an historic misspelling by the developers of the Apache server software often leads to the use of the word "referer" instead). Within the referrers, you should see the URLs of websites that link to your webpages. If you've been working on your search engine visibility, you should certainly see a fair number of referrals from the major search engines. It is those referrals that are our target.

The referrer entry from a search engine will include the search term the user entered that made your webpage appear in the Search Engine Results Pages. They will be encoded to remove <space> characters and other special punctuation, but all in all they're perfectly readable. Take careful note of each one of these. What terms are people using to find your site? You've already been watching your rankings in the search engines, but these entries tell you the search terms that are actually bringing traffic to your site. I have used our logs to discover that there were many people searching for collectibles from Alice in Wonderland. I hadn't realized it was such a popular category until I started watching our log files.

By the same token, are there people coming to your site for your primary keywords? Do you see the search terms for which your website ranks well in the search engines or are your efforts being wasted on search terms that only a few people actually use? This is the kind of information that can help you get the most traffic for your efforts in both constructing your website and your search engine optimization program.

Finally, and most importantly, there are also several services that will tell you what users are entering into the search engines. These tools will show you how often real users searched for the keywords you enter. Both Google and Yahoo! offer keyword research tools through their Pay-Per-Click advertising programs. The best one is the Google AdWords Keyword Tool. These are free tools that are invaluable to finding the best keywords to target for your paid promotional efforts as well as for your search engine optimization. In addition, there are commercial services like WordTracker™ that analyze keyword usage in-depth, including breakdowns of plurarls and other fine details. It's expensive, but if your company really needs to know their market, it can be a great investment.