Give us a call: (800) 252-6164

How Many Googlebot Crawlers Are There? (New data for 2024)

September 13, 2023 | By David Selden-Treiman | Filed in: web-crawler-development.

Wondering how many Googlebot crawlers there are?

Google has 19 Googlebot web crawlers.

What Are Google’s Web Crawlers?

Here’s an overview of what Google’s web crawlers are and what they do:

Crawler NameUse
Common Crawlers
Googlebot SmartphoneUsed for mobile search indexing
Googlebot DesktopUsed for desktop search indexing
Googlebot ImageCrawling image bytes for Google Images and products dependent on images
Googlebot NewsCrawling news articles
Googlebot VideoCrawling video bytes for Google Video and products dependent on videos
Google FaviconFetching favicons; might ignore robots.txt for user-initiated requests
Google StoreBotCrawling certain types of pages like product details, cart pages, etc.
Google-InspectionToolUsed by Search testing tools like the Rich Result Test and URL inspection in Search Console
GoogleOtherOne-off crawls for internal R&D; generic crawler
Special-case Crawlers
APIs-GoogleDelivering push notification messages for Google APIs
AdsBot Mobile Web AndroidChecks Android web page ad quality
AdsBot Mobile WebChecks iPhone web page ad quality
AdsBot DesktopChecks desktop web page ad quality
AdSenseDetermine site content for relevant ad provisioning
Mobile AdSenseDetermine mobile site content for relevant ad provisioning
User-triggered Fetchers
FeedfetcherCrawling RSS or Atom feeds for Google Podcasts, Google News, and PubSubHubbub
Google Publisher CenterProcesses feeds from the Google Publisher Center for Google News landing pages
Google Read AloudFetches and reads out web pages using text-to-speech (TTS)
Google Site VerifierFetches Search Console verification tokens upon user request

What Do The Googlebot User Agent Strings Look Like?

Wondering what the Googlebot useragent strings look like? Here are samples of each one:

Crawler NameSample User Agent
Common Crawlers
Googlebot SmartphoneMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
Googlebot DesktopMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot ImageGooglebot-Image/1.0
Googlebot NewsUses various Googlebot user agent strings (not one specific string mentioned)
Googlebot VideoGooglebot-Video/1.0
Google FaviconMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Google StoreBotMozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
Google-InspectionToolMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
GoogleOtherGoogleOther
Special-case Crawlers
APIs-GoogleAPIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
AdsBot Mobile Web AndroidMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
AdsBot Mobile WebMozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1
AdsBot DesktopAdsBot-Google (+http://www.google.com/adsbot.html)
AdSenseMediapartners-Google
Mobile AdSenseVarious mobile device types (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
User-triggered Fetchers
FeedfetcherFeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Google Publisher CenterGoogleProducer; (+http://goo.gl/7y4SX)
Google Read AloudMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Google Site VerifierMozilla/5.0 (compatible; Google-Site-Verification/1.0)

Source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

David Selden-Treiman, Director of Operations at Potent Pages.

David Selden-Treiman is Director of Operations and a project manager at Potent Pages. He specializes in custom web crawler development, website optimization, server management, web application development, and custom programming. Working at Potent Pages since 2012 and programming since 2003, David has extensive expertise solving problems using programming for dozens of clients. He also has extensive experience managing and optimizing servers, managing dozens of servers for both Potent Pages and other clients.


Tags:

Comments are closed here.

Web Crawlers

Data Collection

There is a lot of data you can collect with a web crawler. Often, xpaths will be the easiest way to identify that info. However, you may also need to deal with AJAX-based data.

Web Crawler Industries

There are a lot of uses of web crawlers across industries. Industries benefiting from web crawlers include:

Legality of Web Crawlers

Web crawlers are generally legal if used properly and respectfully.

Development

Deciding whether to build in-house or finding a contractor will depend on your skillset and requirements. If you do decide to hire, there are a number of considerations you'll want to take into account.

It's important to understand the lifecycle of a web crawler development project whomever you decide to hire.

Building Your Own

If you're looking to build your own web crawler, we have the best tutorials for your preferred programming language: Java, Node, PHP, and Python. We also track tutorials for Apache Nutch, Cheerio, and Scrapy.

Web Crawler Pricing

How Much Does a Web Crawler Cost?

A web crawler costs anywhere from:

  • nothing for open source crawlers,
  • $30-$500+ for commercial solutions, or
  • hundreds or thousands of dollars for custom crawlers.

Factors Affecting Web Crawler Project Costs

There are many factors that affect the price of a web crawler. While the pricing models have changed with the technologies available, ensuring value for money with your web crawler is essential to a successful project.

When planning a web crawler project, make sure that you avoid common misconceptions about web crawler pricing.

Web Crawler Expenses

There are many factors that affect the expenses of web crawlers. In addition to some of the hidden web crawler expenses, it's important to know the fundamentals of web crawlers to get the best success on your web crawler development.

If you're looking to hire a web crawler developer, the hourly rates range from:

  • entry-level developers charging $20-40/hr,
  • mid-level developers with some experience at $60-85/hr,
  • to top-tier experts commanding $100-200+/hr.

GPT & Web Crawlers

GPTs like GPT4 are an excellent addition to web crawlers. GPT4 is more capable than GPT3.5, but not as cost effective especially in a large-scale web crawling context.

There are a number of ways to use GPT3.5 & GPT 4 in web crawlers, but the most common use for us is data analysis. GPTs can also help address some of the issues with large-scale web crawling.

Scroll To Top