a robots.txt file will block spiders from search engines and other sites to gather some datas about your website. The thing that can help you prevent your website from being archived is to use the meta tag robots:
Quote:
<META name="robots" content="nofollow, index">
This meta tag tells robots crawlers, spiders from internet if they have to index the web page, follow the links from the web page, or both. It also tells if they have to do not index or follow links on a web page.
<META name="robots" content="noindex, nofollow">
Robots are not allowed to index the web page or follow the links
<META name="robots" content="index, nofollow">
The robots are allowed to index and follow links
<META name="googlebot" content="noindex, nofollow">
The robot google bot is allew to follow the links but not to index the web page.
<META name="robots" content="all">
Robots are allowed to follow links and index the web page.
<META name="robots" content="none">
Robots are not allowed to follow links and index.
|
from my
website.