How Many Googlebot Crawlers Are There? (New data for 2024)
September 13, 2023 | By David Selden-Treiman | Filed in: web-crawler-development.Wondering how many Googlebot crawlers there are?
Google has 19 Googlebot web crawlers.
What Are Google’s Web Crawlers?
Here’s an overview of what Google’s web crawlers are and what they do:
| Crawler Name | Use |
|---|---|
| Common Crawlers | |
| Googlebot Smartphone | Used for mobile search indexing |
| Googlebot Desktop | Used for desktop search indexing |
| Googlebot Image | Crawling image bytes for Google Images and products dependent on images |
| Googlebot News | Crawling news articles |
| Googlebot Video | Crawling video bytes for Google Video and products dependent on videos |
| Google Favicon | Fetching favicons; might ignore robots.txt for user-initiated requests |
| Google StoreBot | Crawling certain types of pages like product details, cart pages, etc. |
| Google-InspectionTool | Used by Search testing tools like the Rich Result Test and URL inspection in Search Console |
| GoogleOther | One-off crawls for internal R&D; generic crawler |
| Special-case Crawlers | |
| APIs-Google | Delivering push notification messages for Google APIs |
| AdsBot Mobile Web Android | Checks Android web page ad quality |
| AdsBot Mobile Web | Checks iPhone web page ad quality |
| AdsBot Desktop | Checks desktop web page ad quality |
| AdSense | Determine site content for relevant ad provisioning |
| Mobile AdSense | Determine mobile site content for relevant ad provisioning |
| User-triggered Fetchers | |
| Feedfetcher | Crawling RSS or Atom feeds for Google Podcasts, Google News, and PubSubHubbub |
| Google Publisher Center | Processes feeds from the Google Publisher Center for Google News landing pages |
| Google Read Aloud | Fetches and reads out web pages using text-to-speech (TTS) |
| Google Site Verifier | Fetches Search Console verification tokens upon user request |
What Do The Googlebot User Agent Strings Look Like?
Wondering what the Googlebot useragent strings look like? Here are samples of each one:
| Crawler Name | Sample User Agent |
|---|---|
| Common Crawlers | |
| Googlebot Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 |
| Googlebot Desktop | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 |
| Googlebot Image | Googlebot-Image/1.0 |
| Googlebot News | Uses various Googlebot user agent strings (not one specific string mentioned) |
| Googlebot Video | Googlebot-Video/1.0 |
| Google Favicon | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon |
| Google StoreBot | Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 |
| Google-InspectionTool | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 |
| GoogleOther | GoogleOther |
| Special-case Crawlers | |
| APIs-Google | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) |
| AdsBot Mobile Web Android | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 |
| AdsBot Mobile Web | Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 |
| AdsBot Desktop | AdsBot-Google (+http://www.google.com/adsbot.html) |
| AdSense | Mediapartners-Google |
| Mobile AdSense | Various mobile device types (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
| User-triggered Fetchers | |
| Feedfetcher | FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) |
| Google Publisher Center | GoogleProducer; (+http://goo.gl/7y4SX) |
| Google Read Aloud | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 |
| Google Site Verifier | Mozilla/5.0 (compatible; Google-Site-Verification/1.0) |
Source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
David Selden-Treiman is Director of Operations and a project manager at Potent Pages. He specializes in custom web crawler development, website optimization, server management, web application development, and custom programming. Working at Potent Pages since 2012 and programming since 2003, David has extensive expertise solving problems using programming for dozens of clients. He also has extensive experience managing and optimizing servers, managing dozens of servers for both Potent Pages and other clients.
Comments are closed here.