Have you noticed anything different with Google lately? The Webmaster community certainly has, and if recent talk on several search engine optimization (SEO) forums is an indicator, Webmasters are very frustrated. For approximately two years Google has introduced a series of algorithm and filter changes that have led to unpredictable search engine results, and many clean (non-spam) websites have been dropped from the rankings. Google updates used to be monthly, and then quarterly. Now with so many servers, there seems to be several different search engine results rolling through the servers at any time during a quarter. Part of this is the recent Big Daddy update, which is a Google infrastructure update as much as an algorithm update. We believe Big Daddy is using a 64 bit architecture. Pages seem to go from a first page ranking to a spot on the 100th page, or worse yet to the Supplemental index. Google algorithm changes started in November 2003 with the Florida update, which now ranks as a legendary event in the Webmaster community. Then came updates named Austin, Brandy, Bourbon, and Jagger. Now we are dealing with the BigDaddy!

The algorithm problems seem to fall into 4 categories. There are canonical issues, duplicate content issues, the Sandbox, and supplemental page issues

1. Canonical Issues: These occur when a search engine treats www.yourdomain.com, yourdomain.com, and yourdomain.com/index.html all as different websites. When Google does this, it then flags the different copies as duplicate content and penalizes them. Also, if the site not penalized is http://yourdomain.com, but all of the websites link to your website using www.yourdomain.com, then the version left in the index will have no ranking. These are basic issues that other major search engines, such as Yahoo and MSN, have no problem dealing with. Google is possibly the greatest search engine in the world (ranking themselves as a 10 on a scale of 1 to 10). They provide tremendous results for a wide range of topics, and yet they cannot get some basic indexing issues resolved.

2. The Sandbox: This has become one of the legends of the search engine world. It appears that websites, or links to them, are “sandboxed” for a period before they are given full rank in the index, kind of like a maturing time. Some even think it is only applied to a set of competitive keywords, because they were the ones being manipulated the most. The Sandbox existence is debated, and Google has never officially confirmed it. The hypothesis behind the Sandbox is that Google knows that someone cannot create a 100,000 page website overnight, so they have implemented a type of time penalty for new links and sites before fully making the index.

3. Duplicate Content Issues: These have become a major issue on the Internet. Because web pages drive search engine rankings, black hat SEOs (search engine optimizers) started duplicating entire sites’ content under their own domain name, thereby instantly producing a ton of web pages (an example of this would be to download an Encyclopedia onto your website). As a result of this abuse, Google aggressively attacked duplicate content abusers with their algorithm updates. But in the process they knocked out many legitimate sites as collateral damage. One example occurs when someone scrapes your website. Google sees both sites and may determine the legitimate one to be the duplicate. About the only thing a Webmaster can do is track down these sites as they are scraped, and submit a spam report to Google. Another big issue with duplicate content is that there are a lot of legitimate uses of duplicate content. News feeds are the most obvious example. A news story is covered by many websites because it is content the viewers want. Any filter will inevitably catch some legitimate uses.

4. Supplemental Page Issues: Webmasters fondly refer to this as Supplemental Hell. This issue has been reported on places like WebmasterWorld for over a year, but a major shake up around February 23rd has led to a huge outcry from the Webmaster community. This recent shakeup was part of the ongoing BigDaddy rollout that should finish this month. This issue is still unclear, but here is what we know. Google has 2 indexes: the Main index that you get when you search, and the Supplemental index that contains pages that are old, no longer active, have received errors, etc. The Supplemental index is a type of graveyard where web pages go when they are no longer deemed active. No one disputes the need for a Supplemental index. The problem, though, is that active, recent, and clean pages have been showing up in the Supplemental index. Like a dungeon, once they go in, they rarely come out. This issue has been reported with a low noise level for over a year, but the recent February upset has led to a lot of discussion around it. There is not a lot we know about this issue, and no one can seem to find a common cause leading to it.

Google updates were once fairly predictable, with monthly updates that Webmasters anticipated with both joy and angst. Google followed a well published algorithm that gave each website a Page Rank, which is a number given to each webpage based on the number and rank of other web pages pointing to it. When someone searches on a term, all of the web pages deemed relevant are then ordered by their Page Rank.

Google uses a number of factors such as keyword density, page titles, meta tags, and header tags to determine which pages are relevant. This original algorithm favored incoming links and the anchor text of them. The more links you got with an anchor text, the better you ranked for that keyword. As Google gained the bulk of internet searches in the early part of the decade, ranking well in their engine became highly coveted. Add to this the release of Google’s Adsense program, and it became very lucrative. If a website could rank high for a popular keyword, they could run Google ads under Adsense and split the revenue with Google!

This combination led to an avalanche of SEO’ing like the Webmaster world had never seen. The whole nature of links between websites changed. Websites used to link to one another because it was good information for their visitors. But now that link to another website could reduce your search engine rankings, and if it is a link to a competitor, it might boost his. In Google’s algorithm, links coming into your website boost the site’s PageRank (PR), while links from your web pages to other sites reduce your PR. People started creating link farms, doing reciprocal link partnerships, and buying/selling links. Webmasters started linking to each other for mutual ranking help or money, instead of quality content for their visitors. This also led to the wholesale scraping of websites. Black hat SEO’s will take the whole content of a website, put Google’s ad on it, get a few high powered incoming links, and the next thing you know they are ranking high in Google and generating revenue from Google’s Adsense without providing any unique website content.

Worse yet, as Google tries to go after this duplicate content, they sometimes get the real company instead of the scraper. This is all part of the cat and mouse game that has become the Google algorithm. Once Google realized the manipulation that was happening, they decided to aggressively alter their algorithms to prevent it. After all, their goal is to find the most relevant results for their searchers. At the same time, they also faced huge growth with the internet explosion. This has led to a period of unstable updates, causing many top ranking websites to disappear while many spam and scraped websites remain. In spite of Google’s efforts, every change seems to catch more quality websites. Many spam sites and websites that violate Google’s guidelines are caught, but there is an endless tide of more spam websites taking their place.

Some people might believe that this is not a problem. Google is there to provide the best relevant listings for what people are searching on, and for the most part the end user has not noticed an issue with Google’s listings. If they only drop thousands of listings out of millions, then the results are still very good. These problems may not be affecting Google’s bottom line now, but having a search engine that cannot be evolved without producing unintended results will hurt them over time in several ways.

First, as the competition from MSN and Yahoo grows, having the best results will no longer be a given, and these drops in quality listings will hurt them. Next, to stay competitive Google will need to continue to change their algorithms. This will be harder if they cannot make changes without producing unintended results. Finally, having the Webmaster community lose faith in them will make them vulnerable to competition. Webmasters provide Google with two things. They are the word of mouth experts. Also, they run the websites that use Google’s Adsense program. Unlike other monopolies, it is easy to switch search engines. People might also criticize Webmasters for relying on a business model that requires free search engine traffic. Fluctuations in ranking are part of the internet business, and most Webmasters realize this. Webmasters are simply asking Google to fix bugs that cause unintended issues with their sites.

Most Webmasters may blame ranking losses on Google and their bugs. But the truth is that many Webmasters do violate some of the guidelines that Google lays out. Most consider it harmless to bend the rules a little, and assume this is not the reason their websites have issues. In some cases, though, Google is right and has just tweaked its algorithm in the right direction. Here is an example: Google seems to be watching the incoming links to your site to make sure they don’t have the same anchor text (this is the text used in the link on the website linking to you). If too many links use the same anchor text, Google discounts these links. This was originally done by some people to inflate their rankings. Other people did it because one anchor text usually makes sense. This is not really a black hat SEO trick, and it is not called out in Google’s guidelines, but it has caused some websites to lose rank.

Webmasters realize that Google needs to fight spam and black hat SEO manipulation. And to their credit, there is a Google Engineer named Matt Cutts who has a Blog site and participates in SEO forums to assist Webmasters. But given the revenue impact that Google rankings have on companies, Webmasters would like to see even more communication around the known issues, and help with identifying future algorithm issues. No one expects Google to reveal their algorithm or what changes they are making. Rumor on the forum boards speculates that Google is currently looking at items like the age of the domain name, websites on the same IP, and frequency of fresh content. It would be nice from a Webmaster standpoint to be able to report potential bugs to Google, and get a response. It is in Google’s best interest to have a bug free algorithm. This will in turn provide the best search engine results for everyone.

Posted by Rodney Ringler