# All robots will spider the domain User-agent: * Disallow: User-agent: * Disallow: /domain-name/generator/ Disallow: /register/generator/ # BEGIN XML-SITEMAP-PLUGIN Sitemap: http://www.domainnameswebhost.net/sitemap.xml.gz # END XML-SITEMAP-PLUGIN