Welcome to the Question2Answer Q&A. There's also a demo if you just want to try it out.

Most of my site question pages are not crawled by google...why?

0 votes
asked Aug 24 in Q2A Core by Zeeshan

2 Answers

0 votes
answered Sep 3 by LaxmiSharma
selected Sep 7 by Zeeshan
Best answer

Here are some of the issues that may result in not indexing of your pages -

Robots.txt - If your robots.txt file has this line in it; User-agent: * Disallow: /, it means that you are telling the crawler to take a hike and not index ANY of your site's content.

.htaccess - This is an invisible file that lets you toggle visibility in most modern text editors and FTP clients. If this is not configured properly, there may be infinite loops that won’t let your site load at all.

Meta tags - Make sure that the page(s) that's not getting indexed don’t have these Meta tags in the source code: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Sitemaps - Your sitemap isn't updating for some reason, and you keep feeding the old/broken one in Webmaster Tools.

URL parameters – Here you can set URL parameters which tell Google what dynamic links you do not want to get indexed. However, incorrectly configuring parameters can result in pages from your site to be dropped while indexing and therefore it is recommended not to use this parameter unless necessary.

commented Sep 4 by Zeeshan

User-agent: *
Disallow: /user*
Disallow: /register*
Disallow: /login*
Disallow: /ask*
Disallow: /forgot*
Disallow: /chat*
Disallow: /admin*
Crawl-delay: 4

User-agent: Yandex
Crawl-delay: 30

User-agent: SindiceBot
Crawl-delay: 30

User-agent: CCBot
Crawl-Delay: 30

User-agent: wget
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: AhrefsBot
Disallow: /

Sitemap: https://electronics2electrical.com/sitemap.xml
commented Sep 4 by Zeeshan

Options -Indexes
DirectoryIndex index.php
<IfModule mod_rewrite.c>
RewriteEngine On
#RewriteBase /
RewriteCond %{REQUEST_URI} ^(.*)//(.*)$
RewriteRule . %1/%2 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^.*$ index.php?qa-rewrite=$0&%{QUERY_STRING} [L]

RewriteCond %{HTTP_HOST} ^www\.electronics2electrical\.com$
RewriteRule ^/?$ "http\:\/\/electronics2electrical\.com\/" [R=301,L]

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
–1 vote
answered Aug 26 by rahulkukreja
You are not writing content in pages properly.