User-agent: baiduspider Disallow: / User-agent: Flamingo_SearchEngine+(+http://www.flamingosearch.com/bot) Disallow: / User-agent: Whitevector Crawler Disallow: / User-agent: Yandex Disallow: / Crawl-delay: 220 User-agent: Slurp Crawl-delay: 10 # Crawlers that are kind enough to obey, but which we'd rather not have # unless they're feeding search engines. User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-Agent: * Disallow: /report/ Crawl-delay: 10