How to tell if and when your pages are indexed by the search engines.

Search Engine Bytes
Notes, Tips, Questions, & Answers...
aka, topics too short for an article, but too important to leave out!

How to tell if and when your pages are indexed by the search engines.		Can sites be penalized for being listed in too many directories?
How changing your site's registration data might affect your search rankings.		How to quickly find your web pages that are banished to Google's supplemental index.
7 Great Tips for getting those coveted .edu and .gov incoming links!		How overusing keywords in links can cause Google to penalize your pages.
How to tell if and when your pages are indexed by the search engines. Is there a way to know if Google or other search engines have crawled my site's pages recently? Answer: The easiest way to find out is to conduct a search for a specific line of text that is unique to that page. To use one of our own pages as an example: How to Evaluate a Directory Before You Submit a Site ...we might search for the following sentence (surrounded in quotes) that we're reasonably sure is unique to this page: "We considered a number of factors in determining which directories to include." The quotes tell the search engine to search for exactly that set of words in that exact order. As of this writing, that page shows up in each of the major search engines for that search, so we know this page has been indexed. A second important consideration is how recently the page has been indexed. When a page has been recently crawled and indexed, that indicates the search engine places a high priority on maintaining a recent copy of the page (i.e. the page is considered important). On the other hand, if the page hasn't been crawled in a month or more, that indicates the page is not especially important to the search engine due, in many cases, to a lack of incoming links. To see how recently a search engine has indexed a page, click the little cached link that's displayed below the page's listing in the search results. This will retrieve the search engine's copy of the page that is stored on its server. While Google and Microsoft both show the exact date they last indexed the page, Yahoo does not. That's why some webmasters include dynamic code that displays the current date somewhere on the page. That way, when Yahoo grabs a copy of the page, they also grab the date which can then be easily used to determine when Yahoo cached the page. For example, you're running PHP on your server, you could include the following code on your pages: That would display the following: 2007-08-21 08:53:31 Finally, you can also view your server logs or use your web analytics tools to see if and when any specific search engine last crawled your pages. This is the most efficient method when tracking a large number of pages. Our search spider User Agent info page will help you recognize the various spiders in your logs. Can sites be penalized for being listed in too many directories? Your articles suggest that paid links do not make Google happy. On the other hand your book suggests getting a number of links from directories in spite of the fact that most of these links must be purchased. I am wondering, does Google get upset when a site has paid incoming links from the directories? Answer: Although you can get in trouble if you submit to too many directories, Google is fine with directories as a form of paid link for the most part. They even recommend in their terms of service that you submit to the Yahoo Directory. You only run into problems if you're submitting to large numbers of directories, as that can make it look like you're trying too hard to manipulate your search rankings. For new sites, we wouldn't recommend submitting to more than four directories (DMOZ, Yahoo Directory, Business.com, and BOTW) at first. This shows search engines you're committed to your business (most spammers wouldn't spend the money it takes to get listed with these directories) and it provides you with some nice backlinks to help boost your rankings. After that, we'd add roughly one directory listing per 50 'other types' of links. That way your site's incoming link ratio doesn't appear to be overly weighted toward directory listings. When evaluating which directory to get listed with, use the same standards you would with any other site: pay attention to who they are linking to and who's linking to them. That's a topic we cover in detail here: How to Evaluate a Directory Before You Submit a Site The directories listed in our directory list have all been chosen based upon the above standards. You can find that list at: http://www.searchenginenews.com/se-news/directory-master.html How changing your site's registration data can affect your search engine rankings. I am in the process of altering my domain registration and am wondering how changing the admin and technical contact details of my WhoIs info might impact my site's rankings. Although we have not sold the domain or transferred it to a new owner, I am a bit worried that such changes might eradicate the equity from the good links that we've recently acquired. Is my site's good ranking at risk? Answer: Shouldn't be a problem. We've seen plenty of instances where a site was sold to a new company and WhoIs changes didn't effect the site's rankings or backlinks in Google. While it is true that changing WhoIs data can create loss-of-ranking problems for some sites, it's generally confined to sites where it's apparent that an old site was purchased primarily to manipulate another site's link profile (i.e. create a mini-net). In such cases, Google has been known to wipe clean a site's link profile and PageRank. However, WhoIs data changes are only one factor Google takes into consideration in such a case. While it's best not to make frequent changes to your WhoIs registration data, backlink credit doesn't automatically or routinely get reset or wiped clean within Google every time your WhoIs is modified. We strongly suspect that you are in no danger of losing rankings due to the changes you've described. We suggest that you check to see if any PageRank is being passed from those links you've recently acquired. If so, you have a strong indication that Google is counting those links and all is well. How to quickly find your web pages that are banished to Google's supplemental index. I'm concerned that a large number of my site's pages are buried in Google's so-called 'supplemental' index. How do I tell? Answer: You can get a pretty good idea by doing a site search on your domain in Google followed by three asterisks and a negative search for a string of gibberish text. Here's an example; site:www.yourdomain.com *** -jvhwp745herhojd Bear in mind, however, that while this is generally an effective technique, the results fluctuate and Google sometimes returns pages that aren't actually in the supplemental index. Still, this is the best known method of easily finding your site's supplemental pages in bunches. 7 Great Tips for getting those coveted .edu and .gov incoming links! You've mentioned many times that links from .edu and .gov domains are more trusted by Google and therefore lend greater rankings boost. However, it seems harder to get these kinds of links than others. Advice on getting some good trusted .edu and .gov links would be very much appreciated. Answer: Here are some of our favorite ways sites are getting .edu links: sponsoring a student research project, contributions to a department, providing educational resources that are likely to be referenced, getting involved in case studies. We've also had some success working with students who have access to .edu pages. If they're a friend we simply ask them for a link (or if we're an alumnus we get a link from the school's alumnus page). or else find students who will write articles for a site and then link to it from their .edu page. As for .gov's, we've seen instances where sites have received .gov links based on the fact that the company had bid on government contracts. As you are probably aware, Google tends to favor .edu and .gov domains because they appear trustworthy from the search engine point of view. Such sites have generally been around for a long time, have their own trusted backlinks, and they don't generally lend themselves to many spamming techniques. Certainly if it becomes over-abused Google will have to make a correction and assign such favored links less trust on the algorithmic relevance dial. However, for the time being, .edu and .gov links are still relatively difficult to obtain and harder to spam—although we do see on the horizon a slowly growing potential spam problem. In any case, we feel that .edu and .gov links are likely to pass along extra link-juice for a long time to come. Q&A - How overusing keywords in links can cause Google to penalize your pages. Nearly all of my incoming links say mobile phones in the anchor text. I've heard that Google will penalize a site for having too much identical incoming anchor text. I'm getting a bit nervous... Answer: Overuse of keyword anchor text can result in a penalty. There's a factor that's typically referred to as the over-optimization penalty, which began showing an influence on rankings around the time of the Google's Nov 2003 "Florida" update. Anchor text was a big part of this. Leading up to that update, many SEOs had found that Google could be fairly easily manipulated by pointing tons of links at a page with the link anchor text they wanted that page to rank for. Quality of the link didn't particularly matter, as long as you had a lot of them and they all used the same keywords. Google's over-optimization penalty was designed to curb this, and many sites utilizing this technique took a heavy hit in the rankings. Today, anchor text still plays an important role, but it's best to have no more than 30% or so of your incoming links using the same identical anchor text. In other words, it's best if your links look natural, not artificial. Therefore we recommend that you vary your anchor text to use keywords related to your site, modifiers like buy or find (i.e. using find mobile phones instead of just mobile phones) while throwing in a few links that just say click here as well as a few more that use your domain name or page URL. The idea is to make your links look like you weren't directly involved in building them. In the best case scenario, where you're building a good portion of your links through viral marketing and creating a link-worthy site, this won't be an issue. Your natural looking link structure will simply evolve, uh, naturally ;-)