You can’t control what you can’t measure.
That quote is a solid gold nugget. And is also one of the reasons I keep close tabs on Google Webmaster Tools (GWT) for randomtype.ca. So when the site started generating the following results in GWT I started to take notice:
URL Detail Detected ================================================ /blog/listify/ Redirect error Nov 16, 2011 /tag/html/ Redirect error Nov 16, 2011 /tag/net/ Redirect error Nov 16, 2011 /tag/book/ Redirect error Nov 16, 2011 ...
My first thought was that we had made a huge mistake and the site was down. Panic! randomtype.ca was typed manically into the browser… and… and… the page loaded. Phew!
“That’s weird,” I thought “must be an error with Google.” As I’ve proven before, <sarcasm>I’m smarter than Apple</sarcasm>, so I must be smarter than Google! So I left it for a day. But when I came back to check again, I was getting the same results with fresher dates. That concerned me. GWT has a feature called “Fetch as Google Bot” so I tried that. Same results, randomtype.ca was in an infinite loop (redirect error) when fetching via Google Bot.
My next thought was to use curl to see what happened. So I executed the code curl -L http://randomtype.ca the -L switch tells curl to follow redirects. Sure enough it returned the following results:
Running :: curl -L http://randomtype.ca Results :: curl: (47) Maximum (50) redirects followed
So now I’ve got two sources reporting that there’s an infinite loop of redirects happening. This invalidates the first assumption that something is screwing up at Google and leads me to believe that I’ve made a mistake somewhere. The next step is to look at the headers that curl is returning and compare them to a browser’s headers since the browser is successfully loading the page.
curl -I http://randomtype.ca returns the following:
HTTP/1.1 302 Found Server: Apache/2.2 Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Date: Fri, 18 Nov 2011 16:56:38 GMT Location: / Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: wptheme8850c883ba7bf80cad64b51b5d039df9=RANDOMTYPE+Inc.; expires=Sun, 19-Feb-2012 16:56:38 GMT; path=/ Set-Cookie: X-Mapping-cffoajmn=B07F1917A2C28FA3759FA175EBC10612; path=/
And firebug:
HTTP/1.1 200 OK Server: Apache/2.2 Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Content-Encoding: gzip Date: Fri, 18 Nov 2011 16:53:47 GMT X-Pingback: http://randomtype.ca/cms/xmlrpc.php Connection: Keep-Alive Content-Length: 7565
Hmmm… those are two very different headers. During that time I had been working on our site’s .htaccess file so maybe reducing to the default WordPress .htaccess might fix the issue… But it didn’t, still had the same result, including that cookie line. That started to peak my interest, so I ran another curl query to include saving the cookie.
curl -LIc cookie.tmp http://randomtype.ca which returned:
HTTP/1.1 302 Found Server: Apache/2.2 Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Date: Fri, 18 Nov 2011 17:10:55 GMT Location: / Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: wptheme8850c883ba7bf80cad64b51b5d039df9=RANDOMTYPE+Inc.; expires=Sun, 19-Feb-2012 17:10:57 GMT; path=/ Set-Cookie: X-Mapping-cffoajmn=451B32BFF54A8789442A5D320937FD91; path=/ HTTP/1.1 200 OK Server: Apache/2.2 Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Date: Fri, 18 Nov 2011 17:10:57 GMT X-Pingback: http://randomtype.ca/cms/xmlrpc.php Transfer-Encoding: chunked Connection: Keep-Alive
Ahhha! That was it! My investigation into the cookie showed that some of our code for detecting mobile browsers was buggy. And since Google Bot didn’t set the cookie it created an infinite redirect loop. A quick code change and a retest with “Fetch as Google Bot” in GWT confirmed that our code was working.
While it took almost an entire month for those redirect errors to clear out of GWT, we knew that the issue had been solved. Had we not dealt with the issue as swiftly as we did, our entire site may have disappeared off of Google. Clearly not something we’re aiming for!