Question2Answer Q&A - Recent questions tagged robots-txt

where is the error here .... what I should to do?

Sun, 26 Dec 2021 18:23:52 +0000

I have a big problem with my site

I have almost no archiving in Google. There is absolutely no archiving of what I write

What I did, I added a robots.txt file as follows:

********************

User-agent: *

Disallow: /login

Disallow: /login?

Disallow: /forgot

Disallow: /register

Disallow: /register?

Disallow: /admin

Disallow: /users

Disallow: /user

Disallow: /messages

Disallow: /account

Disallow: /search?q=

Disallow: /?show=

sitemap: https://site.com/sitemap.xml

**************************

And now in Google Search Console, in the sitemap section, the following message appears:

https://prnt.sc/24jbkvs

With the following details:

Your sitemap contains too many URLs. Please create multiple sitemaps each with a maximum of 50,000 URLs and submit all sitemaps.

I don't know what to do now I'm frustrated

Now the site is more than a month old and the archiving on Google is completely non-existent

which is the best of robots.txt file 1 or 2 >> please help?

Sun, 08 Aug 2021 18:54:42 +0000

I suffer from very slow archiving

I'm not a developer and I don't understand much about robots.txt

Can you tell me which of the following two files is the best?

1 or 2

thank you in advance

User-agent: Mediapartners-Google

Disallow: /login

Disallow: /ask

Disallow: /forgot

Disallow: /register

Disallow: /questions?sort

Disallow: /admin

Disallow: /*?show=

Sitemap: https://www.yourwebsite.com//sitemap.xml

******************************

User-agent: Googlebot

Allow: /

User-agent: *

Disallow: /

************************************

Allow "tag/*" in robots.txt or not allow ? what are your opinions ?

Mon, 15 Mar 2021 02:03:27 +0000

Allow "tag/*" in robots.txt or not allow ? what are your opinions ?

Robots.txt file can not be modified or removed

Mon, 23 Mar 2020 19:10:30 +0000

After editing the robots.txt file

While accessing the file domaon.com/robots.txt I found that the file itself has been edited but that will not appear by accessing it within the browser

And Google search console reads the old version like the browser

Then after trying to remove it / remove catch files/ removing cloudflare catch nothing happed

The browser still shows the robots.txt file and also Google console while it does not exist

Do i have to block search engines from crawling the other 3 domain versions (Robots.txt

Thu, 28 Mar 2019 08:28:00 +0000

I have all 4 domain versions on Google WMT. 3 of them use a permanent redirect to the preferred.

www and none www > redirects to https none www version