When you first time create your ftp account for uploading your file on server, you can't the Robot.txt file in the ftp. You have to create this file and need to update in your ftp details or server. The Robot.txt file use for telling google or other search engine, which website page you want to index on search engine and which one you want to hide from crawler so they will automatically get de-index. Some time the important page like cpanel, login page of admin available in the ftp file and if the secure data get index on search engine then it will get stolen by spammer.
That's the reason why people use Robot.txt file for re-strutting crawler.