User-agent: * Disallow: / User-agent: Googlebot Disallow: User-agent: AhrefsBot Disallow: / User-agent: Amazonbot # Amazon's user agent Disallow: / # disallow this directory