Robots.txt Files Smiths Internet Services
A robots.txt file can be used to allow certain or all search engines to allow or block content on your
website. Though it is not necessary to have a robots.txt with:
User-Agent: *
Allow: /
It is always better to be safe than sorry and won't do you any harm. You can also get Google to create
the robots.txt file for you via the Google Webmaster Tools.