This script helps you build your robots.txt file providing 22 crawler options to better configure the crawling access to your website.
22 Crawlers to choose from
Multiple Sitemaps supported
Download the file after you finished building it.
Easy to install
Easy to use
Unzip the file Copy the unzipped files to a folder on your histing (create a folder named robots) Accsess the script like this www.yourwebsite/robots/
Follow the USE INSTRUCTIONS BELOW!
1. If your website has a sitemap file(it should have one) or multiple files just enter them in the sitemap input box and separate them by comma (,).The comma separation is very important if you have multiple sitemaps.
2. Select the action (allow/disallow),the crawler bot for wich the action is intended and finally enter the filename for your action(if you want the action to be set for all the files on the website just leave it empty).
3. To add more actions for crawler bots just click the add and apply step 2.
4. Now click Generate Text button to preview the file.
5. Click the Download File button(the file will be downloaded to your PC).
A regular license allows an item to be used in one project for either personal or commercial use by you or on behalf of a client. The item cannot be offered for resale either on its own or as part of a project. Distribution of source files is not permitted.
An extended license allows an item to be used in unlimited projects for either personal or commercial use. The item cannot be offered for resale "as-is". It is allowed to distribute/sublicense the source files as part of a larger project.