Matt Cutts on robots.txt. Basically, use noindex for what you think you’d use robots.txt for. use robots.txt for pagerank sculpting.
About
I am an Austin, Texas based web designer. I have another tumblr - sunburntkamel - it's mostly a feed-scraper.
