Download Link Sleuth and run it on your site
You will get all the links in your site, create robots.txt based on that
On my yet to be published site, I finalized on the following
Also I got the script listed here developed http://www.question2answer.org/qa/15139/fighting-google-panda
Of course, as you can see community did not like the idea showcased in my post but I stand by it as mentioned in the post
Welcome to the Q&A site for Question2Answer.
If you have a question about Q2A, please ask in English.
To report a bug, please create a new issue on Github or ask a question here with the bug tag.
If you just want to try Q2A, please use the demo site.
July 15, 2020: Q2A 1.8.5 released!