robots.txt - Webmaster Forums - Webmaster forum for HTML, PHP, ASP, CSS and more

JR1forums · 05-27-2006, 03:42 AM

I have seen on archive.org that some sites have robots.txt on their server and it prevents archiving. Two questions: How do I setup a robots.txt file on a linux server and will this also block past archives or only future caches of my website (i.e. past will still be available)? Thanks!

Irka · 06-02-2006, 07:18 AM

a robots.txt file will block spiders from search engines and other sites to gather some datas about your website. The thing that can help you prevent your website from being archived is to use the meta tag robots:

Quote:

<META name="robots" content="nofollow, index">

This meta tag tells robots crawlers, spiders from internet if they have to index the web page, follow the links from the web page, or both. It also tells if they have to do not index or follow links on a web page.

<META name="robots" content="noindex, nofollow">
Robots are not allowed to index the web page or follow the links

<META name="robots" content="index, nofollow">
The robots are allowed to index and follow links

<META name="googlebot" content="noindex, nofollow">
The robot google bot is allew to follow the links but not to index the web page.

<META name="robots" content="all">
Robots are allowed to follow links and index the web page.

<META name="robots" content="none">
Robots are not allowed to follow links and index.

from my website.

ApeXX · 06-02-2006, 08:26 AM

Also there is some very good information on meta tags found here: http://searchenginewatch.com/webmast...le.php/2167931.

And if you are having trouble creating the meta tags, you can use a free meta tag builder.
http://vancouver-webpages.com/META/mk-metas.html