Update 2: Just making sure people know that I’m back in the index and Google was very helpful about it. All in all it was a good experience (if still shocking).

Update: Matt Cutts blogged and provided some insight into the process. I want to make sure people know I’m not angry with Google – I broke the rules. It was just a wakeup call for me that Google’s rules are the only ones I care about and I’ll give them whatever they want so I can play.

Yesterday was one of the scariest days I’ve had in a long time and it put in total perspective the power that Google wields. It also made me hope that every single company with an idea about how to improve search, including Microsoft, continues to try and beat Google. I was working on a blog post and was searching for something I’d written before so that I could reference it. Instead of using my blog’s search, I of course used Google’s site feature, like I always do. Only this time, I got nothing:

At first I thought it was just because I was in China and my blog had been blocked. But after asking my Twitter friends to check for me I realized that wasn’t the case. My blog had been completely removed from Google’s index. All of my blog posts were effectively gone to everyone that uses Google, which is basically everyone and especially technology-minded folks that I try to reach as an evangelist.

I was confused, angry, and I felt completely helpless. Everyone uses Google. It was a big referrer to my blog and when I posted tips and tricks for Flash/Flex/AIR I did so knowing that someone could Google the problem and find my blog. My name? No longer my own. The first result was now my blog on ZDNet, which is something I don’t fully control. The other sites which are related to me on the front page were Twitter, and Mike Downey’s Flickr stream – both sites that I have absolutely no control over. My online identity for anyone using Google was now in the hands of domains and brands beyond my reach. It was a scary feeling.

I calmed down, and did some research (using Google of course). I found a blog post that referenced a help@google.com email. After emailing that I got this auto-response:

Thank you for writing to Google. We’d like to assist you, but we only respond to messages submitted through our online contact form. Please visit http://www.google.com/support/ to submit your message, and we’ll get back to you soon. We apologize for any inconvenience, and we look forward to hearing from you.

This is where things get a little spooky and made me realize the full extent of my reliance on Google. I’m not some crazy privacy nut – I live my life very openly on the internet and I’m fine with Google’s business model. But because of how often I rely on Google and how important it is to so many people I’d started treating it like a utility. Just like I have water, electricity, and gas, I have Google. Not so.

Going to the Google Support site and searching led me to this page which says (emphasis mine):

If your site is blocked from our index because it violates our quality guidelines, we may alert you about this using our Webmaster Tools. Simply sign in to our Webmaster Tools, add your site URL, and verify site ownership. The Overview page provides information about the indexing of your site.

I understand that being included in Google’s index is a privilege, not a right. But in order to apply to get my my site re-listed I have to both have a Google account (which conveniently allows me to access all of Google’s wonderful services) and associate my URL with that account. Interesting. So doing all that I was finally able to see what the problem was. I’d been hacked. I had a bunch of spam links that were only showing up to the Google bot.

I have no idea how long they’d been there. I had been fighting spam for a long time but this was worse. I’ve been traveling so I haven’t been able to make sure my blog is totally spam free. I removed the offending spam and applied for reinstatement. Unfortunately according to the message I need to allow several weeks for the request to go through.

Now I realize I should have had better control over my blog. WordPress should have been more secured, I should have been aware of vulnerabilities, and that the responsibility is mine. But I’m a blogger, web developer and evangelist. I don’t know the security side as well as I should and I don’t have time to make sure everything is rock solid. After this experience? Of course. I realize how important it is to conform to Google’s rules.

But I think that’s kind of the problem. I had no warning, no heads up. I like to think I’m a pretty good Netizen – not some SEO firm trying to game the system. My Page Rank is (was) supposedly a 9 for goodness sake (which of course makes me more attractive to spammers). And yet I got totally removed. I use Google because it DOES keep search results free of spam. That’s great. But I realized last night that Google is holding all the cards. They can do whatever the heck they want to. So Microsoft, keep going after search. Get us better results, give webmasters more options. Startups, keep trying to find the weak link. Make Google make itself better. Improve the search experience across the board for everyone – users and webmasters.

Related posts:

1. Google v. Microsoft
2. Would Google Use Firefox as a Rich Internet Application Platform?
3. What Google Trends Says About Rich Internet Applications
4. Google and Rich Media Advertising
5. Google Isn’t Just Using Flash, They’re Using Flex!