Dave's Brain

Crawlers

Crawlers in the week ending 2018-jan-21
CrawlerVisits
SeznamBot/3.2184
DotBot/1.1155
YandexBot/3.0133
MJ12bot/v1.4.8130
bingbot/2.0103
Yahoo! Slurp83
istellabot/t.1.1371
Googlebot/2.157
proximic53
ia_archiver47
AhrefsBot/5.241
Qwantify/2.4w41
Windows NT 10.0; WOW6438
Cliqzbot/2.036
GrapeshotCrawler/2.035
MJ12bot/v1.4.731
SemrushBot/1.2~bl27
coccocbot-web/1.024
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)24
Windows NT 5.1; rv:6.0.220
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)19
ltx71 - (http://ltx71.com/)16
SEOkicks-Robot15
CCBot/2.0 (http://commoncrawl.org/faq/)14
DeuSu/5.0.212
archive.org_bot12
SiteExplorer/1.1b10
MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)10
Macintosh; Intel Mac OS X 10_13_19
Linux x86_648
Win648
Barkrowler/0.7 (+http://www.exensa.com/crawl)7
MSIE 8.06
Steeler/3.56
WOW645
linkdexbot/2.05
ExtLinksBot/1.54
IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)4
MBCrawler/1.0 (https://monitorbacklinks.com)4
compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html4
U4
Go-http-client/1.13
-3
Intel Mac OS X 10.93
Exabot/3.03
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)3
oBot/2.3.13
Dataprovider.com3
Twitterbot/1.03
Windows NT 6.1; WOW643
voltron2
StorygizeBot2
SafeDNSBot (https://www.safedns.com/searchbot)2
GetIntent Crawler (http://getintent.com/bot.html)2
MSIE 9.02
Macintosh; Intel Mac OS X 10_9_22
spbot/5.0.32
yoozBot-2.22
Macintosh; Intel Mac OS X 10_11_41
Macintosh; Intel Mac OS X 10_9_31
Prlog/1.01
X11; Linux x86_641
Macintosh; Intel Mac OS X 10_12_11
Macintosh; Intel Mac OS X 10_12_61
MSIE 7.01
compatible; SpiderLing (a SPIDER for LINGustic research1
AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)1
SearchmetricsBot1
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)1
Findxbot/1.01
Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com1
Windows NT 6.1; rv:38.01
Windows NT 6.3; WOW641
007ac9 Crawler1
compatible; ips-agent1
MojeekBot/0.61
SMTBot/1.01
Yeti/1.11
Seeker v.1 http://LookSeek.com/seeker/1
Macintosh; Intel Mac OS X 10_13_21
ntlk/Nutch-1.9 (Techlimed - Arabic language crawling test)1
MSIE 10.01
Wotbox/2.01 (+http://www.wotbox.com/bot/)1
G-i-g-a-b-o-t1
SurdotlyBot/1.01

Copyright © 2008-2018, dave - Code samples on Dave's Brain is licensed under the Creative Commons Attribution 2.5 License. However other material, including English text has all rights reserved.