I read some related posts on this topic over here but they all seem to have different replies. In addition I am horrible in robots.txt.
I just started Q&A site which is actually on subdomain of my other content site. I created some categories and there are a few questions and answers but I have well over 50 Q&A pages indexed in Google just after a few days. Many categories are still empty but they are indexed.
So what are the best practices for robots.txt file? I don't mind that actual questions and answers will be indexed but tags, categories, users and the like probably should not be in search. Anyone can help?