Welcome to the Question2Answer Q&A. There's also a demo if you just want to try it out.
+3 votes
2.9k views
in Q2A Core by
What will be appropriate robots.txt for question2answer sites?

2 Answers

+2 votes
by

I use the following, which blocks the login page and forgot password page (because they usually contain the "?to=" parameter so you get duplicate content).

User-agent: *
Disallow: /qa/login
Disallow: /qa/forgot


"qa/" here is the subfolder you have Q2A in - if you're using a root domain remove that bit.

by
Thanks for quick answer.

how about this -

User-agent: *

Disallow: /login*
Disallow: /register*
Disallow: /forgot

Sitemap: http://www.xxxxxx.com/sitemap.xml
by
You don't need the * after login and register. Using `Disallow: /qa/login` blocks everything beginning with that URL. Also I'm not sure if a space is allowed between the user-agent and disallow lines.
by
Adding only this two tags will resolve duplicate issue or do I need to add some more?
+1 vote
by
edited by
User-agent: *
Disallow: /login
Disallow: /index.php?qa-rewrite=
Disallow: /ask
Disallow: /forgot
Disallow: /register
Disallow: /questions?sort
Disallow: /chat
Disallow: /admin
Disallow: /activity/*
Disallow: /search?q=
Disallow: /cdn-cgi/
# allow google image bot to search all images
User-agent: Googlebot-Image


Allow: /*


Sitemap: Sitemap address
by
Why block the users pages? And why /questions/*, that's the list of questions which you would want indexed.
by
Thank you
I remove it
by
How about blocking
1. Feedback page (To avoid thin content)
2. Tags (To avoid thin content)
3. Terms and Condition (To avoid duplicate Content)
4. Privacy Policy (To avoid duplicate content)
+Please Add More
...