# go away
User-agent: Ask*
User-agent: Ask Jeeves*
User-agent: Ask Jeeves/Teoma*
User-agent: Baiduspider
User-agent: Baiduspider*
User-agent: BecomeBot
User-agent: BecomeBot*
User-agent: Crawler*
User-agent: EDI
User-agent: EDI*
User-agent: EnaBot
User-agent: EnaBot*
User-agent: Exabot
User-agent: Exabot*
User-agent: Feedfetcher-Google*
User-agent: Gigabot
User-agent: Gigabot*
User-agent: Girafabot
User-agent: Girafabot*
User-agent: Googlebot-Image
User-agent: Googlebot-Image*
User-agent: Googlebot-Mobile
User-agent: Googlebot-Mobile*
User-agent: Hatena Antenna
User-agent: heritrix
User-agent: heritrix*
User-agent: holmes
User-agent: holmes*
User-agent: ia_archiver
User-agent: ia_archiver*
User-agent: ia_archiver-web.archive.org
User-agent: ichiro
User-agent: ichiro*
User-agent: Infoseek SideWinder
User-agent: Infoseek SideWinder*
User-agent: Mediapartners-Google
User-agent: Mediapartners-Google*
User-agent: Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
User-agent: Mozilla/4.0 (compatible; EDI/1.6.5; Edacious & Intelligent Web Robot; Daum Communications Corp., Korea)
User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1; MSIECrawler)
User-agent: Mozilla/4.0 (compatible;)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
User-agent: Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.worio.com/)
User-agent: Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
User-agent: MSIECrawler
User-agent: MSIECrawler*
User-agent: mixi*
User-agent: mixi-mobile-converter/1.0 (http://mixi.jp/)
User-agent: msnbot
User-agent: msnbot*
User-agent: msnbot-media
User-agent: msnbot-media*
User-agent: My Session
User-agent: My Session*
User-agent: NaverBot
User-agent: NaverBot*
User-agent: NutchCVS
User-agent: NutchCVS*
User-agent: NutchCVS/0.7.1 (Nutch running at UW; http://crawlers.cs.washington.edu/; sycrawl@cs.washington.edu)
User-agent: OmniExplorer_Bot
User-agent: OmniExplorer_Bot*
User-agent: OutfoxBot
User-agent: OutfoxBot*
User-agent: PEAR*
User-agent: PEAR HTTP_Request class ( http://pear.php.net/ )
User-agent: page_verifier (http://www.securecomputing.com/goto/pv)
User-agent: RufusBot
User-agent: RufusBot*
User-agent: Shim-Crawler
User-agent: Slurp
User-agent: Slurp*
User-agent: sohu-search
User-agent: sohu-search*
User-agent: Speedy
User-agent: Speedy*
User-agent: Speedy Spider*
User-agent: Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
User-agent: StackRambler
User-agent: StackRambler*
User-agent: SurveyBot*
User-agent: Swooglebot
User-agent: Swooglebot*
User-agent: thumbshots-de-Bot
User-agent: thumbshots-de-Bot*
User-agent: voyager
User-agent: voyager*
User-agent: WWW-Mechanize
User-agent: WWW-Mechanize*
User-agent: Y!J-DSC
User-agent: Y!J-DSC*
User-agent: Y!J-PSC
User-agent: Y!J-PSC*
User-agent: Y!J-SRD
User-agent: Y!J-SRD*
User-agent: Yahoo! Slurp*
User-agent: Yahoo-MMCrawler
User-agent: Yahoo-MMCrawler*
User-agent: Yandex
User-agent: Yandex*
User-agent: Yeti
User-agent: Yeti*
User-agent: ZyBorg
User-agent: ZyBorg*
Disallow: /*/
Disallow: /*
Disallow: /

# permitted is
User-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-agent: Googlebot/2.1 (+http://www.google.com/bot.html)
Allow: /
Disallow: /*/