Prevent Certain Wordpress Folders From Being Crawled By Search Engines

It has been argued that the file structure of a Wordpress blog could leave you open to possible duplicate content penalties. Your posts are viewable on multiple pages of your blog so it could appear to search engines that the data is duplicated and therefore come across as spammy. To avoid that, some bloggers have been creating a robots.txt file and disallowing certain folders from being crawled.

I learned from a post on JohnChow.com which folders to add to the robots.txt file to prevent the search engines from crawling. There has since been a guest post on JohnChow.com that suggests not doing it that way. The argument is that you will end up with less pages in the Google index. The guest post was written from a person who teaches SEO so she could definitely know better than John. Part of her method of avoiding the duplicate content penalty involved using summaries of posts rather than full text. Since I will be using full text for all of my posts I will be following John's method of disallowing certain folders. This is one area though that I will be revisiting at some point. But for now, here is what I do after setting up a new blog...

Create a new text file and call it robots.txt. Add the lines below and save it and upload it to the root of your web site. The folder paths below assume that your blog is installed in the root folder.

User-agent: *
Disallow: /go/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /author/
Disallow: /page/
Disallow: /images/
Disallow: /category/
Disallow: /archives/
Disallow: /trackback/
Disallow: /feed/
Disallow: /*?*

You may not have all of the folders listed above. I created a "go" folder which contains PHP redirects as discussed in the previous post: PHP Redirects And Hiding Affiliate Links. When I investigate this topic further I will create a new post about it.