Welcome to the Question2Answer Q&A. There's also a demo if you just want to try it out.
+2 votes
1.7k views
in Q2A Core by

Hello everyone, i only want to index topics, comments and tags in my website. What kind of robot.txt should i add for this? or could you give me a ready robot.txt file? Thanks..

2 Answers

–1 vote
by

Why not use an existing one, like  that of Stack Exchange's?  

http://physics.stackexchange.com/robots.txt

 

by
Is this from a Q2A website?  I want to restrict access to this subdirectory:
register?to=

However this is not included there.
+1 vote
by
You can use this file.

https://meta.question2answer.info/robots.txt
(change sitemap url)
keep monitoring all unwanted bots which are slowing down your site and then add to robots.txt list. It totally depends on your site. check server log /var/log/httpd/access_log file for bot info. Dont block useful bots like Google, Bing etc.
by
Can you suggest a robot.txt for q2a integrated with wordpress?
by
You can use this one https://question2answer.info/robots.txt

If you are having q2a and wordpress on same domain then combine these two robots.txt file.
...