How do I configure the robots.txt file? SiteBuilder Pro

Procédure

What is the Robots.txt file?

This is a file that tells the various search engines what they can and cannot index. Before indexing a website, search engines consult the robots.txt file.

Where can I find the robots.txt file on a website?

The "robots.txt" file is located at the root of your website. In principle, this file is generated automatically when your website is created, but it is possible to modify it to suit your needs.

How do I activate the robots.txt file in SiteBuilder Pro?

To activate the"robots.txt" file on your website, you first need to connect to your website administration.

1. Click on the"Settings" tab in the SiteBuilder Pro toolbar. Then click on"Settings" to open the control panel.

2. Go to the"Robots.txt" tab and tick the"Enable robots.txt" box.

By default, if you don't configure anything else in your"robots.txt" file, it will contain three lines of code, as follows:

• User-agent: this is used to indicate the robot affected by this rule. Each engine has its own robot, each with a different name. The * character defines that the rule applies to all.
• Allow: This means"authorisation". The / character is often associated with this line to define that all the URLs on your site can be indexed.
• Sitemap: This is another file containing the tree structure of your website. All the pages on your site are included. By default, the"sitemap.xml" file is created automatically at the root of your website.

How do I configure the robots.txt file on SiteBuilder Pro?

You can modify the "robots.txt" file at any time. You may have specific needs to indicate to the various search engines. For example, if you want to prevent a page on your site from being indexed, you need to indicate the following in the "robots.txt" file

User-Agent: googlebot Disallow: /page.html Sitemap: http://votre-domaine.com/sitemap.xml

It is possible to exclude several pages from indexing. Just add as many Disallow lines as you want:

User-Agent: googlebot Disallow: /page.html Disallow: /page2.html Disallow: /page3.html Sitemap: http://votre-domaine.com/sitemap.xml

If you have a large"robots.txt" file, it is often useful to comment out your lines so that you can find your way around:

User-Agent: googlebot Disallow: /page1.html # test page of my website Disallow /tmp # contains temporary files

How do I address specific search engines?

In order to address specific search engines, you will need to use the following configuration:

User-Agent: googlebot Disallow: /folder/ User-agent: Bingbot Disallow: /folder2/

How do you exclude your website from indexing?

It is often advisable, when you are in the middle of creating your website, toexclude indexing temporarily, while you finish setting up the various pages. To prevent your site from being indexed, you need to insert this code:

User-agent: * Disallow: /

How do I define an exception in the robots.txt file?

You may want to block indexing of an entire folder except for a file that is important to you. To do this, you must first block indexing of the entire folder and then allow indexing of one or more files in the same folder.

User-agent: googlebot Disallow: /blog/ Allow: /blog/page.html

Conclusion

You are now able to activate and configure the"robots.txt" file on your site created with SiteBuilder Pro. The"robots.txt" file will tell the various search engines which pages of your site you want to index.

Don't hesitate to share your comments and questions!