IntroductionEvery website that is interested in getting as many visitors as possible should be using both
If you don't have a
User-agent: * Disallow:
To use you simple place this file at the root of your webserver. So if your website is at
If there are certain parts of your site that you don't want them to visit you can add a
User-agent: * Disallow: /data/ Disallow: /scripts/
You can even disallow all robots from accessing anywhere on your site with this
User-agent: * Disallow: /
One final command that you can use that relates to the next section of this page is the '
User-agent: * Disallow: SITEMAP: http://www.advancedhtml.co.uk/sitemap.txt
For more information on robots.txt files go to http://www.robotstxt.org/.
Whereas robots.txt files are usually used to ask robots to avoid a particular part of your site, a sitemap is used to give the robot a list of pages that it is welcome to visit.
By giving the search engine a sitemap you can (hopefully) increase the number of pages that it indexes. As well as telling the search engine the URLs of your pages, the sitemap can also tell the robots when the page was last modified, the pages priority, and how often the page is likely to be updated.
There are two main sitemap formats. The simplest is a simple text file listing the full URLs of all your pages. The second is an XML file which can provide a lot more information. For this site I use a simple text file. Here is a shortened version of what it looks like.
The file format shouldn't need much explanation. It is just a text file with the list of URLs. I save it as sitemap.txt and put it on my webserver at
The XML version of the sitemap format looks like that shown below. I would recommend that you generate them using a sitemap generation tool rather than trying to hand code them. Search for sitemap generation tools on Google. I used http://www.xml-sitemaps.com/ to create the below snippet.
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> <url> <loc>http://www.advancedhtml.co.uk/</loc> <priority>1.00</priority> <changefreq>weekly</changefreq> </url> <url> <loc>http://www.advancedhtml.co.uk/advancedhtml.htm</loc> <priority>0.80</priority> <changefreq>weekly</changefreq> </url> <url> <loc>http://www.advancedhtml.co.uk/tables.htm</loc> <priority>0.80</priority> <changefreq>weekly</changefreq> </url> <url> <loc>http://www.advancedhtml.co.uk/colours.htm</loc> <priority>0.80</priority> <changefreq>weekly</changefreq> </url> </urlset>
You should call your XML sitemap '
If you add a reference to your sitemap to your robots.txt file then it should be found by the search engines automatically. However you can take a more active role in the sitemap submission process by using tools from Google, Microsoft and Yahoo. You can read more about these tools on my Website Analytics page.
Advanced HTML Home
Copyright © 1997 - 2017
Hosted by 1&1