Computer Forensics

I have a client which offers data recovery and computer forensics services. I am very happy to say that I have not been in need of their service (knock on wood).

We recently ran into a very strange problem with their site dropping completely out of google’s index. My first instinct was that the seo company they were previously working with had gone a little to far, and was responsible for getting them banned. However, upon further investigation, I learned that their site had vanished because of a problem with their robots.txt file. Apparently there was a problem with their custom 404 page which was raising a 500 server error. Google interpreted this differently than if their robots.txt simply didn’t exist at all. Instead google assumed that it must not be safe to crawl this site period, since they are unable to adhere to the robots rules.

The weirdest part of all, is from what I can tell, nothing has been touched on this site in years, and their 404 page has probably always behaved this way. Google must have recently changed something on their end which cause this unfortunate series of events.

The good news is, it only took me a couple minutes to fix the offending robots file and google had reindexed the site and was bringing in traffic again within a few days.