The robots.txt file does not have anything to do with keywords. When a robot, spider, etc. crawls your website, it is supposed to first look for the robots.txt file before it crawls anyfiles. You can specify which bots you do or do not want searching your site. You can also disallow them from accessing spefic pages or directories. For example, maybe you have a private area where members must login to access, you can specify that you do not want any robots to access this directory. All you do to create it is create a new text doc if your on windows use like notepad, so there is no formatting. Here is an example of a robots.txt file that allows all robots and disallows them from the cgi-bin.
Code:
User-agent: *
Disallow: /cgi-bin
Upload this file to the top level publicly available directory, so that it can be accessed by typing in
http://yourdomain.com/robots.txt
For an example check out the
White House's robots.txt file.