The following info is intended to assist you in identifying the search engine spiders (also known as robots, bots or crawlers) that visit your site. Details are based on information you can obtain by viewing your site's visitor log reports.
Use the User Agent to identify a particular search engine in your site's visitor logs. Use the Robots.txt identifier if you'd like to block that spider from accesssing parts of your site using a Robots.txt file.
There are other search engine spiders, of course, but they either get their results from one of the search engines listed here, or are too small to bother with. To see where the major search engines get their results, see our search engine relationship chart (updated monthly).
For information on blocking any of these spiders using the Robots.txt exclusion standard, see http://www.robotstxt.org/wc/exclusion.html
| Ask | Google Images | |
| Yahoo | LookSmart | MSN |
| |
| | ||
| Company | Ask | |
| User Agent | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml) | |
| Robots.txt Identifier | | User-agent: Teoma |
| Details | http://about.ask.com/en/docs/about/webmasters.shtml | |
| | ||
| Company | ||
| User Agent | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Robots.txt Identifier | User-agent: Googlebot | |
| Details | http://www.google.com/bot.html | |
| | ||
| Company | ||
| User Agent | Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) | |
| Robots.txt Identifier | User-agent: Googlebot-Mobile | |
| Details | http://www.google.com/support/webmasters/bin/answer.py?answer=35308 | |
| | ||
| Company | Google Images | |
| User Agent | Googlebot-Image/1.0 | |
| Robots.txt Identifier | User-agent: Googlebot-Image | |
| Details | http://www.google.com/support/webmasters/bin/answer.py?answer=35308 | |
| | ||
| Company | Google Adsense | |
| User Agent | Mediapartners-Google/2.1 | |
| Robots.txt Identifier | User-agent: Mediapartners-Google | |
| Details | http://www.google.com/bot.html Note: This and other Googlebots share crawled pages with each other to reduce the amount of crawling Google is required to do for their different services. | |
| | ||
| Company | LookSmart | |
| User Agent | Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com) | |
| Robots.txt Identifier | User-agent: ZyBorg | |
| Details | Powers LookSmart's Wisenut.com http://www.WISEnutbot.com | |
| | ||
| Company | Microsoft / MSN | |
| User Agent | msnbot/1.0 (+http://search.msn.com/msnbot.htm) msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) msnbot-news/1.0 (+http://search.msn.com/msnbot.htm) msnbot-products/1.0 (+http://search.msn.com/msnbot.htm) | |
| Robots.txt Identifier | User-agent: msnbot User-agent: msnbot-media User-agent: msnbot-news User-agent: msnbot-products | |
| Details | MSN has not updated their page http://search.msn.com/msnbot.htm yet to officially address the specific bots they are using. If you wish to block a specific bot in your robots.txt file use the associated bot specific name as the User-agent | |
| | ||
| Company | Yahoo | |
| User Agent | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Robots.txt Identifier | User-agent: Slurp | |
| Details | http://help.yahoo.com/help/us/ysearch/slurp/ | |
| | ||
| Company | Yahoo | |
| User Agent | Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) | |
| Robots.txt Identifier | User-agent: YahooSeeker/M1A1-R2D2 | |
| Details | http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html | |
Nuk ka komente:
Posto një koment