SEO Case Study: The disappearing, de-indexed website
Ever had the feeling when you were a kid and something really terrible happens? You get caught cheating on a test. Girl rejects you in front of a bunch of friends. Mom walks in on you…well, you get the drift.
But you know that terrible terrible feeling inside your stomach?
Well, that is the feeling I got the other day when I stumbled upon a Webmaster Tools account for a client, who also happens to be a good friend of mine who we are helping out.
Immediately my mind starts going crazy.
Our team deploys and we try to get to the root of the problem. Our best men are on the job.
Diagnosing a De-indexed Site
We checked robots.txt right away to see if there were any issues in there.
We noticed some weirdness in the robots.txt and wasn’t positive of a few things so I just deleted the entire robots.txt (backed it up of course) so I could check that out later. We could always add it back and if there are crawl issues I would just rather that have Google open access than have a whack robots.txt.
After that I checked the .htaccess file just in case some tom-foolery happened in there. Nothing odd in there, just the standard WordPress permalink rewrites so I moved on from there.
Dropped back over to Google Webmaster Tools to check the index status and things got way worse.
translation: you are screwed
Overnight we basically went from 422 indexed pages to 6. The fact that there was 6 rather than 0 worried me even more because that seemed like it was some sort of algorithmic action rather than a development error of some point.
At this point we know that the loss in traffic / impressions is most likely due to the fact that Google has de-indexed a great portion of our site.
So we start going through the whole site.
We stripped down the whole site to keep things simple. We also took this time to delete a ton of transient files / directories in FTP that were left over from a previous install. At least 10k files and about 900Mb of data deleted.
Let me go try and fetch a few things to see what is going on in that department.
More nonsense and now I’m really starting to get nervous. Google basically hates us now.
I didn’t need to do anymore digging at this point. I knew we had a major issue that needed to be figured out.
The sitemap looked great and there was no errors. I started checking plugins for any misconfigured settings when wouldn’t you know it…
We found the problem
There was an option that was causing all of our URI’s to become deindexed.
So I navigate to these “privacy settings” which takes me to a 404. Apparently WordPress moved these settings to Settings > Reading a while back.
A few seconds later I click my way over to this portion of the dashboard and wouldn’t you know it, the old box was ticked.
So yea WordPress, pretty please with sugar on top discourage Google from indexing this site.
The deadly part of this setting is that it is really easy to miss if you aren’t looking for it.
Rather than WordPress inserting a line into robots.txt, it inserts a robots noindex,follow into the header site wide.
<meta name='robots' content='noindex,follow' />.
From here we unchecked this menacing check box and verify the site is no longer in noindex mode.
Things are looking up from here.
We go back to Webmaster Tools to re-fetch a few pages to see what happens.
So yea, it looks like that is what it was.
Very small and sometimes accidental mistakes can cause massive amounts of damage.
We still have no idea how this happened. It could have been one of us who checked the box by accident, it could have been the client, or the former SEO company even perhaps.
That said, this is just another argument for making sure to change your passwords on a regular basis.
I know this might not seem like a major problem to outsiders and yes it was a really obvious fix but sometimes that is just the way it is. Sometimes the smallest errors cause the most problems.