May 24, 2022

A robots.txt Guide For Newspaper Publishers

While more and more companies are entering into a paid search battle to get more traffic it seems that newspapers are trying to do just the opposite. This is especially the case in Belgium. First there was the court case that ruled against Google claiming that they couldn’t index a Belgian newspaper. Now, as TechDirt points out, it looks like they’re going after Yahoo.

In an effort to help out, I’m now going to tell all newspapers how to get their sites de-listed from ALL search engines without having to hire a lawyer.

Step 1: Create a new text file called robots.txt The easiest way to do this is to click on start in the lower left corner, click run. Now, type in “notepad robots.txt”. A Windows prompt will come up saying “cannot find file, would you like to create one?”. Select Yes.

Step 2: Enter the following text:

User-agent: *
Disallow: /

Don’t forget to save this file.

Step 3: Upload this file to your server. You may have to talk to somebody from your tech team for help with this. Another way is to go to start, run and this time type in ftp://Username:[email protected] (of course, replace username, password and yoursite with the actual details. Then, you can just drag and drop the robots.txt file into the window. Congratulations! You’ve now blocked Google AND Yahoo from visiting your website, and it didn’t even cost you any retainer fees.

Of course you’ve also cut off millions of potential visitors, which begs the question “Why are you even putting your articles online if you don’t want anybody to see them?”

About Ryan Jones

Ryan Jones is an SEO from Detroit. By day he works as a manager of SEO & Analytics at SapientNitro where his team performs SEO for Fortune500 clients. By night he's either playing hockey or attempting to take over the world with his own websites - which he would have already succeeded in doing had it not been for those meddling kids and their dog. The views expressed here have not been paid for and belong only to Ryan, not any of his employers or clients. Follow Ryan on Twitter at: @RyanJones, add him on Google+ or visit his personal website:


  1. This is a test of my new comment anti spam stuff


  1. […] More and more newspapers don’t want their articles to be found or read online by anybody, and they’re willing to sue Google to do so. Here’s how to do it without all the lawsuits. Via […]