In my previous post, I let you know that I had finally achieved zero supplemental pages in the Google Search results for my site. For those that don’t know, you can determine how many pages you have in the Supplemental results by using the following search string.
site:geekinparadise.com *** -ndsfoiw
Replace geekinparadise.com with your own domain name, of course. If you want to see the total number of pages indexed by Google for your site, drop everything after your domain name. I’ve read that you can use a different string after the (-), but most places I did research on used that string. As it stands now, geekinparadise.com has the same number of pages when searching both strings, meaning no supplemental pages.
Now, as promised, here is my robots.txt. Feel free to let me know if something isn’t necessary or overkill as I’m far from being a web genius.
User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: */feed/
Disallow: /trackback/
Disallow: /cgi-bin/
Disallow: /category/
User-agent: Googlebot
Disallow: /category/
Disallow: */feed/
Disallow: /feed
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /*?*
Disallow: /*?
Disallow: /z/
Disallow: /wp-*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Allow: /wp-content/uploads/
User-agent: Googlebot-Image
Allow: /*
User-agent: Mediapartners-Google
Allow: /*
Sitemap: http://geekinparadise.com/sitemap.xml
There’s probably some overkill and whatnot in there, but it’s worked out fine as far as I can tell.
Tags: WordPress Robots.txt Supplemental
