Description:

It is impossible to know how many pages Google correctly indexed on the site. The search engine does not maintain a database by URLs.

The approximate number of pages in the search results will be shown by the site operator, which we rely on. The number can be distorted by pages that are banned from indexing in robots.txt, but hit the search results because of external links to them.

The “Indexing Status” section in Google Search Console will show a slightly more correct number, but this data can also be distorted by the filters’ application.

Google scans websites to find infected resources, phishing pages, and other issues that degrade the search results and user experience. With this information, the search engine warns users about unsafe sites. If a site is deemed dangerous, Google might downgrade or remove it.

Additional Information:

• Google Webmaster Panel
• Google Safe-browsing

For successful indexing of the page by search bots, the HTTP response code of the server must be 200.

Additional Information:

• [Checking the server response of the internal pages of the site](/tools/http-response/
• List of status codes

Description

Our bot checks site availability by sending an HTTP request. If the server returns a status code other than 200, the bot cannot access the site content. This can be caused by IP blocking, bot protection (e.g. Cloudflare, QRATOR), server errors, or misconfiguration.

How to allow the PR-CY bot to access your site?

1. Cloudflare: create a Firewall Rule — allow IP range 87.228.72.128/25 or User Agent "PR-CY".
2. robots.txt: make sure it doesn't block the bot. Add: User-agent: PR-CY-BOT / Allow: /
3. Server rules: check server rules (.htaccess, etc.) to ensure they don't block IP range 87.228.72.128/25.

Check your site's accessibility for our bot using the PR-CY Bot Tester.

Description:

A robots.txt file is a list of restrictions for search robots or bots that visit a site and crawl information there. Before crawling and indexing your site, all robots go to the robots.txt file and look for rules.

The robots.txt file is located in the root catalogue of the site. It must be available on the URL: pr-cy.io/robots.txt

There are several reasons to use a robots.txt file on your site:

1. to remove duplicate content;
2. to hide unwanted information;
3. to limit the indexing speed.

Description:

The llms.txt file is a standard that helps large language models (LLMs) better understand and use your site's content. The file is located in the root directory of the site and contains structured information about the resource.

Having an llms.txt file improves the interaction of AI systems with your site and can enhance the quality of responses generated based on your content.

Description:

A Sitemap file is a file with information about the pages on the site to be indexed. With this file you can:

• tell search engines which pages of your site need to be indexed;
• find out how often the information on the pages is updated;
• understand which pages are most important to index.

Additional Information:

• Official Documentation

For the promotion of commercial sites, the confidentiality of the exchange of information between the server and visitors is important. This increases the loyalty of potential customers to the resource, increases the level of trust, affects the conversion and growth of positions in the search results for almost all requests.

Description:

Technically, domains with www and without www are two different resources; search engines index and rank them separately, and links will have different weights. This can lead to:

• Decrease in search results
• Filter, because a search engine may mistake one site for a duplicate of another;
• Problems with authorization on the site and other functionality that uses cookies;

The problem is solved by a 301 redirect and an indication of the main mirror to the search engines. From the website promotion point of view, a domain without www is better, because it is not a third-level domain, and its length will always be shorter.

Download speed directly affects behavioral factors: the faster the download, the fewer bounces. The Google robot visits slow sites less often. This affects the effectiveness of promotion, such sites are rarely indexed. Loading speed is one of the main factors in Google ranking.

Description:

While requesting a non-existent page, the server should return a 404 error which is"page not found."

If the server is set up incorrectly and a 200 error returns, then the page exists. In this case, search engines may index all pages of the site with errors.

Set up the site in such a way that while requesting non-existent pages a 404 response code "page not found" or a response code 410 "page deleted" appear.

Description:

During requesting a non-existent page, the server displays a standard page with a 404 error. For the users’ convenience we recommend making a unique 404 page and adding a backlink to the site on it.

Don't forget to prolong your domain name. It’s better to involve auto-renewal with your registrar. After the end of the domain registration, there is a chance to lose access to the domain.

IP addresses ever found on the site. And also the sites that have the same IP address.