28 May 2010

When Google locked the door

When Google locked the door

Google logo with a door (election doodle)
This is the story of how Google, for a period of three years, locked me out of their groups service, how I eventually found my way back in, and what it cost me.

Yeah, I guess this is a bit off topic regarding to software development. However these days many of us store important data and value in services like Google's. Services with terms like: "Google may stop providing the Services to you at Google’s sole discretion, without prior notice to you". I guess I never took it too seriously, as the companies would probably get seriously bad PR, if they did something like that. Deleting emails for billions. My error was forgetting the case where you are the only person being locked out.

I my case the lock wasn't from email, but from Google Groups. Not as critical as email would have been, but still, well, rather inconvenient. The lockout meant that I was unable to manage the PyChess mailing list. I was unable to fight the, at that time, increasing spam level; and more importantly I couldn't reply anybody in my community.


December 2006 I sat up a Google group for the PyChess project. It made it possible to have a dual web/mailing-list forum for the project, and as we already used Google for a lot of things it was an easy choice. It was easy as magic to set up and start using, and so we did, heavily.
After about a years use, I started feeling a problem. When I wrote mails to the list, no one would answer. And when I answered other peoples post, they seamed to ignore them and press for new answers. As I tried to check the online group to see what was happening, I got a 403 Forbidden error. After a short while I realised that this error was given for any page one the groups.google.com sub domain.

My first thought was naturally to clear all cookies and history and sign off. The last thing helped, but as soon as I signed in to my group the error was back: Username. Password. Enter. 403.
I tried cookie cleaner tools, other browsers and even other peoples computers, but every single time, the result was the same:


Slowly (maybe I was too naive) I began to realize how things were set. I had been locked out. The question was why? and what now? I tried to recall if any action of mine could be linked to the time of the lockout, but no. I hadn't really used other groups than my own, and I had certainly not participated in threads more aggressive than the typical comp.lang.python. I checked my 'recent posts' to see if there had been abuse, but no.
My guess was some kind of database error. If it was an intended ban I hope they'd at least have sent a warning as well as giving a more informative error message. All I saw was the black on white 403.

What now? I couldn't go to the usual fora. The problem was too specific. I had to get to somebody who could actually help me. (Naivly?) I went to the Google homepage looking for something like 'Support'. The page was easy to scan, and I found nothing. There was however an 'About Google' link, which after a few steps lead me to a page named Help.
The help page has been improved a bit since then, but the result is the same. It is made to keep you away from actual supporters and feed you canned or crowd sourced material. Where ever you click, you are sent to a support paged hosted at - you guessed it - groups.google.com. No go.
After a few hours of finding an actual email adress I gave up and went to sleep.

The next morning problems had found me. A group member wrote to me about a spammer, who I, as the admin, was the only one who could kick. I had banned a spammer once before, and it wasn't hard. Well - if I actually had access to the control panel.

I had to go back to the support. I went through the hassle of creating a new account, so I could post to the help forum. I expressed my problems and asked if anybody knew of a way to get to Google.

I didn't get the impression that Google were reading the forum, and as I had expected nobody could really help me.
A guy gave me a link to a help page, that looked a bit different from the others. The page was mostly for people who had forgotten their passwords, which was clear when I had to specify an alternate write back email address. As the error message expressed it: "Please enter an email address, that you have access to". Sigh. But it was my best shot.
As I was well aware, that they were probably busy, I took care to specify my problem as specific as possible, what I thought caused it, and how they could help me. It should be an easy fix.


After a year I received an answer:
10/31/09
Hello,

We've received your report that you're having trouble accessing a Google
product. After completing our investigation, we've found a violation of
our Terms of Service: http://google.com/accounts/TOS -OR- product-specific
program policies:
http://www.google.com/support/accounts/bin/answer.py?answer=147806. As a
result, we're unable to grant you access to this product.

Regards,
The Google Team

Arg. This led nowhere. I instantly asked back for more details, and after a month of no reply I tried with a long begging letter, but neither were ever answered.


At this time I was near the break point. Should I just change to another group service? That would require me to manually migrate any wanted archive material, rebuild the community, and it would take a way some of the nice integration between the services we used.
At that time migration was the only option on
However at that time it was the easiest solution, competing only with one point on my list: Visit the nearest Google office. In Norway.

My laziness postponed the decision for a couple of months, but then suddently something happened. For a long time I had wanted the pychess.org domain name for the project, but it was controlled by 'Pensylvania Youth Chess'. They never had actually got up a website though, and now they gave up and transfered me the domain.
Being a great Google fan, I sat the domain up with Apps for Domains. When I first logged in I saw this:


My eyes were flickering for a bit. Then staring right below the 'Try Premier Edition Free' link: '24/7 Phone support'. What I had been looking for for so long was suddently standing in front of me in a nice blue box.
I knew that Premier Edition would give me Google groups directly for my domain, however the project being voluntary work, I couldn't affort paying 50$ a month per member. But did it matter? No! The trail period should give me plenty of support time, so I could go back using my current group.
After staring amazed at nothing for about a minute I clicked the button.

By an act of magic, the support I had been looking for for 3 years was suddently at my fingertips. All that stood between my and accessing groups.google.com now, was to convince the support team, that unblocking my personal Google account was important for my use of Apps for Domains.
I figured that if I had wanted to switch to the Google groups in Premier Edition, I would've wanted to transfer emails from my old group. For this I would need access. (I don't think there actually is a 'transfer archive to new group' option, but it was the best I could come up with).
At first the support was a bit confused and tried to help me by enabling the groups feature for my domain. (Which was of course already enabled), but after a few mails where I kept insisting, I got a mail from another supporter:

Hello Thomas,

Thank you for confirming your error message. I've manually reset your old admin account, which should resolve this issue.

Please let me know if you still encounter errors, and I will continue troubleshooting.

Sincerely,

[Name]
Enterprise Support


Bing! This was all the magic I needed. Now it worked.
I suppose it took the second supporter about a minute to reset the account, given it is not something they do too often. For me it had been a three years splinter.

This concludes the tutorial: How to get access back to Google.
As the services are free of charge, I never really expected any support options. However I also never really expected the need for them. When things failed, I saw no way to buy the missing support, and the friendly facade suddenly seamed like a tall dark wall.
Perhaps the grief of this issue is in its rareness, but how can we know how often this kind of thing happens? If any admin can lock you all out by a sloppy click, and give you no option to defend yourself, then it is bound to happen once in a while.


Postscript:
A month after my sucess, my bank acc was billed 50$ for a renewal of Premier Apps. This happened regardless of the checked option 'Do not automatically renew my subscription'. I wrote the support in detailes, and they replied to me, that if I didn't want it to happen again, I should check 'Do not automatically renew my subscription'. Doh.

Now that I had paied expensively for the support, I also wanted to use it. So I gave it my best show and wrote back. Eventually luck was on my side:

Hello Thomas,

Thank you for your message. It is our policy that there is not refunds after a charge has been processed. However, under your special circumstance, I have authorized to issue you a refund of your Premier Renewal charges.

You should see the refund in your bank account if a few days. In addition, your account was downgraded to the Standard Edition.

I hope this resolves your issue. Please contact me if you required additional assistance.

Sincerely,

[name]
Enterprise Support

Thank you Google! I never lost trust in you.

Update: Slashdotted


Thanks for all the great comments at http://yro.slashdot.org/story/10/05/31/0151202/Where-Do-You-Go-When-Google-Locks-You-Out . Especially thanks to IamTheRealMike for his insight into Googles strategy and thoughts on these kinds of things.