Webmaster Forums - Webmaster forum for HTML, PHP, ASP, CSS and more - View Single Post

JR1forums · 05-27-2006, 03:42 AM

I have seen on archive.org that some sites have robots.txt on their server and it prevents archiving. Two questions: How do I setup a robots.txt file on a linux server and will this also block past archives or only future caches of my website (i.e. past will still be available)? Thanks!

05-27-2006, 03:42 AM	#1 (permalink)
JR1forums Junior Member Join Date: May 2006 Posts: 3	robots.txt I have seen on archive.org that some sites have robots.txt on their server and it prevents archiving. Two questions: How do I setup a robots.txt file on a linux server and will this also block past archives or only future caches of my website (i.e. past will still be available)? Thanks!