How Many Googlebot Crawlers Are There? (New data for 2024)

September 13, 2023 | By David Selden-Treiman | Filed in: web-crawler-development.

Wondering how many Googlebot crawlers there are?

Google has 19 Googlebot web crawlers.

What Are Google’s Web Crawlers?

Here’s an overview of what Google’s web crawlers are and what they do:

Crawler Name	Use
Common Crawlers
Googlebot Smartphone	Used for mobile search indexing
Googlebot Desktop	Used for desktop search indexing
Googlebot Image	Crawling image bytes for Google Images and products dependent on images
Googlebot News	Crawling news articles
Googlebot Video	Crawling video bytes for Google Video and products dependent on videos
Google Favicon	Fetching favicons; might ignore robots.txt for user-initiated requests
Google StoreBot	Crawling certain types of pages like product details, cart pages, etc.
Google-InspectionTool	Used by Search testing tools like the Rich Result Test and URL inspection in Search Console
GoogleOther	One-off crawls for internal R&D; generic crawler
Special-case Crawlers
APIs-Google	Delivering push notification messages for Google APIs
AdsBot Mobile Web Android	Checks Android web page ad quality
AdsBot Mobile Web	Checks iPhone web page ad quality
AdsBot Desktop	Checks desktop web page ad quality
AdSense	Determine site content for relevant ad provisioning
Mobile AdSense	Determine mobile site content for relevant ad provisioning
User-triggered Fetchers
Feedfetcher	Crawling RSS or Atom feeds for Google Podcasts, Google News, and PubSubHubbub
Google Publisher Center	Processes feeds from the Google Publisher Center for Google News landing pages
Google Read Aloud	Fetches and reads out web pages using text-to-speech (TTS)
Google Site Verifier	Fetches Search Console verification tokens upon user request

What Do The Googlebot User Agent Strings Look Like?

Wondering what the Googlebot useragent strings look like? Here are samples of each one:

Crawler Name	Sample User Agent
Common Crawlers
Googlebot Smartphone	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
Googlebot Desktop	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot Image	Googlebot-Image/1.0
Googlebot News	Uses various Googlebot user agent strings (not one specific string mentioned)
Googlebot Video	Googlebot-Video/1.0
Google Favicon	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Google StoreBot	Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
Google-InspectionTool	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
GoogleOther	GoogleOther
Special-case Crawlers
APIs-Google	APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
AdsBot Mobile Web Android	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
AdsBot Mobile Web	Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1
AdsBot Desktop	AdsBot-Google (+http://www.google.com/adsbot.html)
AdSense	Mediapartners-Google
Mobile AdSense	Various mobile device types (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
User-triggered Fetchers
Feedfetcher	FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Google Publisher Center	GoogleProducer; (+http://goo.gl/7y4SX)
Google Read Aloud	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Google Site Verifier	Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

Source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

David Selden-Treiman

David Selden-Treiman is Director of Operations and a project manager at Potent Pages. He specializes in custom web crawler development, website optimization, server management, web application development, and custom programming. Working at Potent Pages since 2012 and programming since 2003, David has extensive expertise solving problems using programming for dozens of clients. He also has extensive experience managing and optimizing servers, managing dozens of servers for both Potent Pages and other clients.

Tags: Google Web Crawler

Comments are closed here.

Web Crawlers

Data Collection

There is a lot of data you can collect with a web crawler. Often, xpaths will be the easiest way to identify that info. However, you may also need to deal with AJAX-based data.

Web Crawler Industries

There are a lot of uses of web crawlers across industries. Industries benefiting from web crawlers include:

Legality of Web Crawlers

Web crawlers are generally legal if used properly and respectfully.

Development

Deciding whether to build in-house or finding a contractor will depend on your skillset and requirements. If you do decide to hire, there are a number of considerations you'll want to take into account.

It's important to understand the lifecycle of a web crawler development project whomever you decide to hire.

Building Your Own

If you're looking to build your own web crawler, we have the best tutorials for your preferred programming language: Java, Node, PHP, and Python. We also track tutorials for Apache Nutch, Cheerio, and Scrapy.

Hedge Funds & Custom Data

Custom Data For Hedge Funds

Developing and testing hypotheses is essential for hedge funds. Custom data can be one of the best tools to do this.

There are many types of custom data for hedge funds, as well as many ways to get it.

Implementation

There are many different types of financial firms that can benefit from custom data. These include macro hedge funds, as well as hedge funds with long, short, or long-short equity portfolios.

Leading Indicators

Developing leading indicators is essential for predicting movements in the equities markets. Custom data is a great way to help do this.

Web Crawler Pricing

How Much Does a Web Crawler Cost?

A web crawler costs anywhere from:

nothing for open source crawlers,
$30-$500+ for commercial solutions, or
hundreds or thousands of dollars for custom crawlers.

Factors Affecting Web Crawler Project Costs

There are many factors that affect the price of a web crawler. While the pricing models have changed with the technologies available, ensuring value for money with your web crawler is essential to a successful project.

When planning a web crawler project, make sure that you avoid common misconceptions about web crawler pricing.

Web Crawler Expenses

There are many factors that affect the expenses of web crawlers. In addition to some of the hidden web crawler expenses, it's important to know the fundamentals of web crawlers to get the best success on your web crawler development.

If you're looking to hire a web crawler developer, the hourly rates range from:

entry-level developers charging $20-40/hr,
mid-level developers with some experience at $60-85/hr,
to top-tier experts commanding $100-200+/hr.

GPT & Web Crawlers

GPTs like GPT4 are an excellent addition to web crawlers. GPT4 is more capable than GPT3.5, but not as cost effective especially in a large-scale web crawling context.

There are a number of ways to use GPT3.5 & GPT 4 in web crawlers, but the most common use for us is data analysis. GPTs can also help address some of the issues with large-scale web crawling.