It is time to set the record straight. That is, from an SEO perspective anyway.
For many years I’ve heard a lot of different opinions, rumors, and misinformation about 404’s when it comes to SEO. Today rather that share my opinion, I’d like to highlight some facts that have been made available from Google’s own documentation and a few other resources.
Perhaps you are one of those people that has been misguided over the years. Perhaps you are one of the ones that have been trying to get this message across, and no one will listen. Whatever the case may be, the time is now to speak loud and clear about 404’s and how they fit with search engine optimization.
Let’s start with a powerful quote from Google’s own, Gary Illyes:
Whoever came up with the idea that having 404s gives a site any sort of penalty, you’re wrong. Utterly wrong.
— Gary Illyes (@methode) August 6, 2015
If 404’s are OK then why does Google warn me about them?
Great question, excellent question! How did you get so good at questions?
Tis true, Google does give stern warnings for 404 errors. In fact Google just warned me the other day that our agency had quite a few of them.
Also, before I go any further I’d like to clarify that in most cases when I use the term “404” I do so synonymously with “410” (gone).
The problem Google has with 404’s isn’t with the 404 in and of itself, the problem is with how the 404 was produced.
For instance, if Google is crawling your about page which has a link to a page that no longer exists, that is bad for your user experience which in turn is bad for Googlebot.
OR if your server is misconfigured and Google lands on it, causing a 404 it is also bad for user experience and Googlebot.
OR if your website is down, forgot to pay hosting bill, domain, DNS, CDN whatever issues that could cause a 404.
Again, while 404’s won’t cause a penalty in and of itself, they can/might cause problems for your users, and could cause Googlebot some crawling problems.
It is worth noting that the 404’s that you find via Google Search Console are / can be different than ones that you find on your own. For example, my personal 404 monitor for this website picked up this 404:
but it was not reported in Google Search Console. This is because Google never “saw” this 404. It was generated as a result of someone entering the URL manually into the browser (or a hacker via terminal) thus it never entered Google’s hemisphere.
Common misconceptions about 404’s
I’ll admit, in the beginning of my SEO career I fell victim to some of the SEO underground propaganda being spread throughout the community.
One of the most common misconceptions for SEO’s who have 404 errors is to “301 them” or even worse to “301 them to the homepage” by default. While in some situations it might be ok to 301/302 a 404 to another page or the home page, it should by no means be the default action.
To clarify, here is what Google says:
“Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Such pages are called soft 404s, and can be confusing to both users and search engines.”
Another common issue is that your server might be acting up. This could happen as a result of a number of issues such as a faulty plugin, misconfigured framework (such as WordPress, Magento, etc) server issues, or a misconfigured .htaccess file.
404’s can tell a story
Like many error codes and server status messages, 404’s can tell a story about what is going with your situation. For instance let’s say you see a bunch of 404’s for something that looks like this:
http://www.example.com/admin.php http://www.example.com/login.php http://www.example.com/user-login.html
^ most likely an attacker fingerprinting your server looking for your admin or login page.
^ most likely a webmaster with bad spelling habits
404's all generated on 4/14/2016 at 4:05pm http://www.example.com/home http://www.example.com/about-us http://www.example.com/services http://www.example.com/contact http://www.example.com/mission http://www.example.com/blog http://www.example.com/blog-post-1
^ most likely means your server / domain went down
When not to fix 404’s
By default most of the time with a “one off” 404 the best course of action is to do nothing. If a page was removed, keep it removed.
One of the first things I do when Google sends me a warning about a 404 is checked the “Crawl Errors” page, select the error then checked the “linked from” tab. This will show what pages are linking to the 404.
Only redirect the link if there is an appropriate substitution. For instance let’s say that Moz.com just deleted a page and replaced it with a “better or newer” version. In that case, the old URL or 404 should be redirected to the new page. (In all reality, they probably would have kept the URL/slug/permalink and not deleted anything.)
But what if a webmaster made a typo in a hyperlink and fixed it the next day? Do they have to create a 301 just for this case? In my opinion, no as long as the link is fixed.
There are many other times when a 404 can remain “unfixed” such as an old URL or a type. For e.g. let’s make up some random URL’s on legit websites:
If you see a URL in Google Search Console that isn’t linked from any other pages, and seems to be a typo, it is OK to mark it as “fixed.”
If you follow both of those URL’s, they’ll take you to a 404 page because they do not exist. Is Google or Inbound going to “fix” them? Probably not, because it was just a typo.
If there are external sites linking to the page, in that case I would recommend 301’ing the old link to the new one, or to a similar resource if you must.
There are so many examples I could give, another common example is a website that does an upgrade from a plain HTML website to a WordPress website. This will cause 404’s to be generated automatically, so in this case we’d recommend doing 301’s for any pages that has a lot of links to it or until it finishes re-indexing in the SERPs. Case by case.
If you see that you have quite a bit of broken links / 404’s on your website, it might be time to do a 404 / broken link audit on your website. For medium to large sites, we find it is probably a good idea to do this at the very least a few times per year.
Just know this: most 404 / broken link scenarios are different from website to website. While I’ve been a webmaster for the better part of my life, I ‘m still learning new things and making great revaluations all of the time.
If you have anything to add, please feel free to do so within the comments!