What to do when you're dropped from Google's index

About one week after we were featured in the Financial Post, I received a phone call that made my heart drop into my stomach. Basically, here's the gist of what was said: "we searched for you yesterday and found your site ranked #1, but I'm searching now and I don't see it anywhere in Google".

OK, at first I didn't panic, I just assumed it was Google showing a different personalized result. But after I searched on all of our top keywords and even validated through 3rd party rank checkers, I was aghast to realize that we were no longer #1. In fact, we were no longer in the index at all!

I quickly ran through all possible scenarios in my head. Were we being penalized for paid links? No, we hadn't purchased a link since the whole debacle with Google's new stance on the issue. Was Google manually de-indexing us after the exposure in the Financial Post? I don't see how that's possible, we only promote Google-approved SEO techniques. Was it sabotage? Could someone have proxied our site or gotten us removed?

I decided to wait a day or two to see if there was a change. But the next day the results were just as bad - the only thing indexed was the blog, and it too was slipping. Even searching for our unique company name "Convurgency" provided poor results, we weren't even #1 for that, and the link was to the blog. Still, there was no explanation of what happened.

One of the most curious events during this time was that there was a PR update, and we were even bumped up to a PR5. This despite our nonexistence in the index. So I knew something fishy was up. With the help of some connections, we were able to receive some confirmation that we had not been manually penalized. And yet still, we were nowhere to be found.

The answer came about a week later. I was checking Webmaster Tools for the umpteenth time, and finalyl saw an error message on the diagnostics page: " We can't currently access your home page because of an unknown problem." OK, this doesn't tell me why they can't see my site, but at least explains why they've dumped me!

I did some digging in Google Groups, and found that others who've seen that error discovered that their hosting provider was blocking the Googlebot! I contacted support, and sure enough, that's what happened. I asked them to unblock it immediately, and asked for an explanation. Here's what they said: "It was likely blocked because it was looping on one of our accounts when crawling and causing the server to function slowly."

So, I'm left with the following possibilities:

• one of the websites on the server was indeed causing the Googlebot to loop
• one of my competitors was spoofing the Googlebot and hammering my site

Which was it? We'll probably never know. However, we continue to watch the Webmaster Tools like a hawk, and if it happens again we'll need to move to a dedicated server.

Thankfully we're back at #1. But it was a harsh lesson to learn for those couple of weeks of obscurity!