Why Isn'y My Website in the Search Engines?
Another of the most common complaints I see on
Google Webmaster Help Forum is
"I can't find my site in Google!" Submission alone is not sufficient to get a website indexed. It
will get your website's main page crawled, but that's just a first step. It's good to remember that the search engines are
eager to include new websites in their index, so there is usually a simple explaination why a new site hasn't been
indexed. This Search Engine Optimization Tip will show you the most common problems and their solutions.
Reasons Why A Website Isn't Indexed
- No Inbound Links To The Site - You need at least one good quality link from another
website to be included (ie. "indexed") in the major search engines. This is the main reason
why new websites are not indexed. Start by submitting your site to a good quality directory like
The Open Directory Project, JoeAnt,
or one of the others I recommend in my Building
Links SEO Tips article. Ask other websites whose main topic is related to yours to link to your site. A good quality
link comes from a prominent page on a high quality website. Forum signatures and blog comments are not likely
to help your ranking much, but they can help get your site indexed.
- Site Blocked By The robots.txt File - The robots.txt file is a plain text file that controls
how search engines access your site. Many website designers block the search engines from crawling a site while it's under
construction, and then forget to remove that blocking instruction when the design in completed. So check your robots.txt
file in Google's Webmaster Tools, where a tool is provided
that will test your robots.txt file. You'll also find information there on how to use your robots.txt file properly.
- Site Blocked By robots <META> Tag - Similarly, some web designers will
insert a robots <META> tag set to "noindex" on a website's main page while it's in the testing
stage. The "noindex" robots <meta> tag will prevent a page from being indexed, or will remove
it if it has previously been indexed. Remove the tag and you should recover in a short time.
- Server And DNS Problems - It's rare, but not unheard of, for server problems to
be severe enough to prevent the search engines from accessing a site. Sometimes hosting services
have automatic controls that see search engine crawlers (a.k.a. "robots", a.k.a. "spiders")
as malicious and block them. Use the "Fetch As Googlebot" tool in Google's Webmaster Tools console
to make sure that Googlebot (Google's crawler) can access your site. Extremely slow response times can
also prevent indexing. An improperly configured Domain Name Server (DNS) can sometimes cause problems
as well. Check your domain's DNS at intoDNS, and if their system
reports problems, consult your hosting service about it immediately. If you don't get a satisfactory
response, you should move to a new hosting service.
- Frame-Forwarded Domain - Some free and low-cost services have webmasters create a
<frameset> page as the main page of a domain name package, but the website's actual content resides on
a completely different domain. The major search engines see such a website as having no content of its own
and will not index it. In this situation, the search engines (epsecially Google) will often index the content
at the domain where it truly resides. The solution is to buy a conventional hosting service, or at least use
a reliable free host, so that your content is seen to be on your own domain.
- Invalid/Poorly Constructed Webpages - The search engines' crawlers are very tolerant of
errors in the HTML mark-up of webpages, but it is possible for mark-up and syntax errors to prevent indexing in extreme
cases. Valid HTML mark-up will not only help insure that your site is indexed properly, it will
also insure that your users see what you wanted them to see when they visit your site. Check your
site's HTML with the W3C's HTML Validator and correct any errors
- Previously-Owned Domain Banned/Penalized - If your domain name was previously
owned and was banned for violating the Guidelines in the past, the search engines might continue to exclude it
from the index. You can check the history of the domain name by using the Wayback
Machine operated by archive.org. Look at the old pages shown in the Wayback Machine for signs of a poor
quality website. If the website used to be about pornography, drugs, or other often-abused topic, or if the site
looks like it was created by copying the content of other websites, there's a good chance it was banned. The
solution is to file a Reconsideration Request in Google's Webmaster Tools console, or use the Email Support
form in Bing's Webmaster Center. Tell them that you are the new owner of a domain you believe has been
penalized, and that you intend to adhere to their guidelines.
As I say, the search engines are anxious to include your website so they can show it in
the search results. You just have to make sure that they see signals of quality (like an inbound link!),
and that your site behaves properly. Of course, being indexed is not the same as ranking well, but it's a
very necessary first step in your Search Engine Optimization and Marketing program.
This page was last updated on June 13, 2015
Need More Help?
You'll find more SEO Tips on the menu on the right side of this page.
You can also contact me with your SEO questions.
If you can't fix your website search engine problems on your own,
my Search Engine Optimization Services
can give your website what it needs to get your business' fair share of search engine traffic quickly, without disturbing your website's design, and without breaking your budget.
Call Richard L. Trethewey at Rainbo Design in Minneapolis today at 612-408-4057 from 9:00 AM to 5:00 PM Central time
to get started on an affordable website design package or search engine optimization program for your small business today!
In writing these SEO tips, I'm often reminded of a pearl of wisdom that my high school computer programming
teacher passed on from one of his teachers, "Computers are high-speed idiots!" Remember that, and don't let them get
under your skin.