# Robots.txt file to Allow, Disallow Search engines, Crowlers to index your website User-agent: Googlebot Allow: / # Next line will disallow all # User-agent: * # Disallow: / # Specific engines User-agent: sitecheck.internetseer.com Disallow: / User-agent: Openbot Disallow: / User-agent: Slurp Disallow: / User-agent: turnitinbot Disallow: /