web spider list, arama motoru crawler listesi, web spider names

Alexa=ia_archiver
Altavista=Scooter
Anzwers=AnzwersCrawl
Ask=Teoma
Atomz=Atomz
Boitho=boitho.com
Entireweb=Speedy Spider
Exalead=Exabot
Excite=ArchitextSpider
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=GigaBot
Google=Googlebot
Google-Image=Googlebot-Image
Yahoo!=Yahoo! Slurp
Infoseek=Ultraseek
Inktomi=Slurp
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Microsoft Research=MSRBOT
MSN=MSNBOT
NetSeer=Teemer
noXtrum=noxtrumbot
Searchme=Charlotte
Seznam=SeznamBot
Snap=Snapbot
Voila=VoilaBot
Walhello=appie
WISEnut=ZyBorg
Yahoo=Yahoo! Slurp

Yahoo=Yahoo Slurp
Inktomi=Slurp
MSN=MSNBOT
Sogou=sogou web spider
Entireweb=Speedy Spider
Voila=Voila.fr
Ask.com=Ask Jeeves
Teoma=TeomaAgent
Wisenut=Zyborg
NorthernLight.com=Gulliver
Excite=Architext spider
AltaVista=Mercator
Crawler.de=Crawler
Infoseek=InfoSeek sidewinder
Lycos=Lycos_Spider_(T-Rex)
Search Hippo=Fluffy the Spider
Infoseek=Ultraseek
Looksmart=MantraAgent
Webcrawler.com=WebCrawler
Twiceler=Twiceler-0.9
Naver.com=Yeti/

.htaccess blok listesi

RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX [OR]
RewriteCond %{HTTP_USER_AGENT} Yeti


ABACHOBot
AbiLogicBot
Accoona-AI-Agent
AnyApexBot
Arachmo
B-l-i-t-z-B-O-T
Baiduspider
BecomeBot
Bimbot
BlitzBOT
boitho.com-dc
boitho.com-robot
btbot
Cerberian Drtrs
ConveraCrawler
cosmos
DataparkSearch
DiamondBot
EmeraldShield.com WebBot
envolk[ITS]spider
EsperanzaBot
Exabot
FAST Enterprise Crawler
FAST-WebCrawler
FDSE robot
FindLinks
FurlBot
FyberSpider
g2crawler
Gaisbot
genieBot
Gigabot
Girafabot
hl_ftien_spider
htdig
ia_archiver
ichiro
IRLbot
IssueCrawler
Java
Jyxobot
LapozzBot
Larbin
libwww-perl
LinkWalker
lmspider
lwp-trivial
mabontland
MJ12bot
Mnogosearch
mogimogi
MojeekBot
Morning Paper
msnbot
MSRBot
MVAClient
NetResearchServer
NG-Search
nicebot
noxtrumbot
Nusearch Spider
NutchCVS
obot
oegp
OmniExplorer_Bot
Orbiter
PageBitesHyperBot
polybot
Pompos
Psbot
PycURL
Python-urllib
RAMPyBot
RufusBot
SandCrawler
SBIder
Scrubby
SearchSight
Seekbot
semanticdiscovery
Sensis Web Crawler
SEOChat::Bot
Shim-Crawler
ShopWiki
Shoula robot
silk
Snappy
sogou spider
Speedy Spider
Sqworm
StackRambler
SurveyBot
SynooBot
Teoma
TerrawizBot
TheSuBot
Thumbnail.CZ robot
TinEye
TurnitinBot
updated
VoilaBot
Vortex
voyager
VYU2
webcollage
Websquash.com
wf84
WoFindeIch Robot
Xaldon_WebSpider
yacy
Yahoo! Slurp
Yahoo! Slurp China
YahooSeeker
YahooSeeker-Testing
yoogliFetchAgent
Zao
Zealbot
zspider
ZyBorg

örnek robots.txt
User-agent: ABACHOBot
Disallow: /


User-agent: AbiLogicBot
Disallow: /


User-agent: Accoona-AI-Agent
Disallow: /


User-agent: AnyApexBot
Disallow: /


User-agent: Arachmo
Disallow: /


User-agent: B-l-i-t-z-B-O-T
Disallow: /


User-agent: Baiduspider
Disallow: /


User-agent: BecomeBot
Disallow: /


User-agent: Bimbot
Disallow: /


User-agent: BlitzBOT
Disallow: /


User-agent: boitho.com-dc
Disallow: /


User-agent: boitho.com-robot
Disallow: /


User-agent: btbot
Disallow: /


User-agent: Cerberian Drtrs
Disallow: /


User-agent: ConveraCrawler
Disallow: /


User-agent: cosmos
Disallow: /


User-agent: DataparkSearch
Disallow: /


User-agent: DiamondBot
Disallow: /


User-agent: EmeraldShield.com WebBot
Disallow: /


User-agent: envolk[ITS]spider
Disallow: /


User-agent: EsperanzaBot
Disallow: /


User-agent: Exabot
Disallow: /


User-agent: FAST Enterprise Crawler
Disallow: /


User-agent: FAST-WebCrawler
Disallow: /


User-agent: FDSE robot
Disallow: /


User-agent: FindLinks
Disallow: /


User-agent: FurlBot
Disallow: /


User-agent: FyberSpider
Disallow: /


User-agent: g2crawler
Disallow: /


User-agent: Gaisbot
Disallow: /


User-agent: genieBot
Disallow: /


User-agent: Gigabot
Disallow: /


User-agent: Girafabot
Disallow: /


User-agent: hl_ftien_spider
Disallow: /


User-agent: htdig
Disallow: /


User-agent: ia_archiver
Disallow: /


User-agent: ichiro
Disallow: /


User-agent: IRLbot
Disallow: /


User-agent: IssueCrawler
Disallow: /


User-agent: Java
Disallow: /


User-agent: Jyxobot
Disallow: /


User-agent: LapozzBot
Disallow: /


User-agent: Larbin
Disallow: /


User-agent: libwww-perl
Disallow: /


User-agent: LinkWalker
Disallow: /


User-agent: lmspider
Disallow: /


User-agent: lwp-trivial
Disallow: /


User-agent: mabontland
Disallow: /


User-agent: MJ12bot
Disallow: /


User-agent: Mnogosearch
Disallow: /


User-agent: mogimogi
Disallow: /


User-agent: MojeekBot
Disallow: /


User-agent: Morning Paper
Disallow: /


User-agent: msnbot
Disallow: /


User-agent: MSRBot
Disallow: /


User-agent: MVAClient
Disallow: /


User-agent: NetResearchServer
Disallow: /


User-agent: NG-Search
Disallow: /


User-agent: nicebot
Disallow: /


User-agent: noxtrumbot
Disallow: /


User-agent: Nusearch Spider
Disallow: /


User-agent: NutchCVS
Disallow: /


User-agent: obot
Disallow: /


User-agent: oegp
Disallow: /


User-agent: OmniExplorer_Bot
Disallow: /


User-agent: Orbiter
Disallow: /


User-agent: PageBitesHyperBot
Disallow: /


User-agent: polybot
Disallow: /


User-agent: Pompos
Disallow: /


User-agent: Psbot
Disallow: /


User-agent: PycURL
Disallow: /


User-agent: Python-urllib
Disallow: /


User-agent: RAMPyBot
Disallow: /


User-agent: RufusBot
Disallow: /


User-agent: SandCrawler
Disallow: /


User-agent: SBIder
Disallow: /


User-agent: Scrubby
Disallow: /


User-agent: SearchSight
Disallow: /


User-agent: Seekbot
Disallow: /


User-agent: semanticdiscovery
Disallow: /


User-agent: Sensis Web Crawler
Disallow: /


User-agent: SEOChat::Bot
Disallow: /


User-agent: Shim-Crawler
Disallow: /


User-agent: ShopWiki
Disallow: /


User-agent: Shoula robot
Disallow: /


User-agent: silk
Disallow: /


User-agent: Snappy
Disallow: /


User-agent: sogou spider
Disallow: /


User-agent: Speedy Spider
Disallow: /


User-agent: Sqworm
Disallow: /


User-agent: StackRambler
Disallow: /


User-agent: SurveyBot
Disallow: /


User-agent: SynooBot
Disallow: /


User-agent: Teoma
Disallow: /


User-agent: TerrawizBot
Disallow: /


User-agent: TheSuBot
Disallow: /


User-agent: Thumbnail.CZ robot
Disallow: /


User-agent: TinEye
Disallow: /


User-agent: TurnitinBot
Disallow: /


User-agent: updated
Disallow: /


User-agent: VoilaBot
Disallow: /


User-agent: Vortex
Disallow: /


User-agent: voyager
Disallow: /


User-agent: VYU2
Disallow: /


User-agent: webcollage
Disallow: /


User-agent: Websquash.com
Disallow: /


User-agent: wf84
Disallow: /


User-agent: WoFindeIch Robot
Disallow: /


User-agent: Xaldon_WebSpider
Disallow: /


User-agent: yacy
Disallow: /


User-agent: Yahoo! Slurp
Disallow: /


User-agent: Yahoo! Slurp China
Disallow: /


User-agent: YahooSeeker
Disallow: /


User-agent: YahooSeeker-Testing
Disallow: /


User-agent: yoogliFetchAgent
Disallow: /


User-agent: Zao
Disallow: /


User-agent: Zealbot
Disallow: /


User-agent: zspider
Disallow: /


User-agent: ZyBorg
Disallow: /