robots.txt for SEO

Question

What will be appropriate robots.txt for question2answer sites?

Answer 1

I use the following, which blocks the login page and forgot password page (because they usually contain the "?to=" parameter so you get duplicate content).

User-agent: *
Disallow: /qa/login
Disallow: /qa/forgot

"qa/" here is the subfolder you have Q2A in - if you're using a root domain remove that bit.

Comments

Thanks for quick answer.

how about this -

User-agent: *

Disallow: /login*
Disallow: /register*
Disallow: /forgot

Sitemap: http://www.xxxxxx.com/sitemap.xml

---

You don't need the * after login and register. Using `Disallow: /qa/login` blocks everything beginning with that URL. Also I'm not sure if a space is allowed between the user-agent and disallow lines.