# No need to index cruft, may contain legally interesting things User-agent: * Disallow: /cruft # BecomeBot is for shopping sites User-agent: BecomeBot Disallow: /