Dave's Brain

Crawlers

Crawlers in the week ending 2018-may-20
CrawlerVisits
BLEXBot/1.0214
DotBot/1.1171
MJ12bot/v1.4.8168
YandexBot/3.0122
istellabot/t.1.13117
SeznamBot/3.2116
Daum/4.1106
bingbot/2.0106
Googlebot/2.168
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)58
AhrefsBot/5.256
The Knowledge AI51
GrapeshotCrawler/2.050
CCBot/2.0 (http://commoncrawl.org/faq/)41
Cliqzbot/2.039
proximic38
Qwantify/2.4w36
ia_archiver32
ltx71 - (http://ltx71.com/)29
Windows NT 10.0; WOW6429
coccocbot-web/1.027
Windows NT 5.1; rv:6.0.222
Linux x86_6422
python-requests/2.18.422
SemrushBot/1.2~bl19
MauiBot (crawler.feedback+wc@gmail.com)18
Windows NT 5.1; rv:6.017
MJ12bot/v1.4.716
Windows NT 6.1; WOW6411
SemrushBot/1.0~bm11
MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)10
Baiduspider/2.09
compatible;contxbot/1.08
Macintosh; Intel Mac OS X 10_13_18
Clickagy Intelligence Bot v28
Steeler/3.57
Exabot/3.07
Win645
Dataprovider.com4
U4
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)4
archive.org_bot4
MSIE 10.04
linkdexbot/2.04
amd64 Linux 4.4.0-119-generic4
Intel Mac OS X 10.93
A6-Indexer3
YisouSpider3
SiteExplorer/1.1b3
Macintosh; Intel Mac OS X 10_10_13
spbot/5.0.33
SafeDNSBot (https://www.safedns.com/searchbot)2
Go-http-client/1.12
uCrawler/1.02
IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)2
SEOkicks-Robot2
yoozBot-2.22
Intel Mac OS X 10.102
Scrapy/1.5.0 (+https://scrapy.org)2
Windows NT 6.3; WOW642
MSIE 9.02
ExtLinksBot/1.52
oBot/2.3.12
compatible; ips-agent1
Trident/7.01
NetSeer crawler/2.01
nutch-1.4/Nutch-1.41
MSIE 7.01
compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot1
Windows NT 6.1; rv:28.01
X11; Linux x86_641
Plukkie/1.61
Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com1
Hatena::Fetcher/0.01 (master) Furl/3.061
Macintosh; Intel Mac OS X 10_13_41
eright/1.01
Prlog/1.01
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36/1
Windows NT 6.1; rv:38.01
WOW641
Mappy/1.01
SurdotlyBot/1.01

Copyright © 2008-2018, dave - Code samples on Dave's Brain is licensed under the Creative Commons Attribution 2.5 License. However other material, including English text has all rights reserved.