In order for all search engines to be directed to your sitemap.xml file in your website's route directory, you can add a line to your robots.txt file that specifies the URL of your sitemap.
The robots.txt file would then look something like this:
User-agent: *
Sitemap: http://www.mywebsite.com/sitemap.xml
Disallow: /landingpages/
Disallow: /campaigns/ppc.html
Examples of a folder (landingpages) and a file (ppc.html, in the 'campaigns' folder) that search engines are being instructed not to spider / include in their index are also included.
The * tells all search engines to obey the robots text file rules.
Note that if you have a website that you do not wish indexed by any search engine, you would include one 'Disallow' line:
Disallow: /
You can add as many 'Disallow' lines as necessary.
If you have not yet created a sitemap file for your website, it is wise to do so. You should submit it to Google via their Webmaster Central service.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment