Give us a call: (800) 252-6164

Archives: Web Crawler

How Many Googlebot Crawlers Are There? (New data for 2023)

September 13, 2023 | By admin

Wondering how many Googlebot crawlers there are? Google has 19 Googlebot web crawlers. What Are Google’s Web Crawlers? Here’s an • Read More »

How Much Does a Web Crawler Cost?

March 5, 2023 | By admin

Discover the true cost of a web crawler for your business needs. From open-source to commercial solutions, find the best fit and ROI for your budget.

All About XPaths & Web Crawlers

February 17, 2023 | By admin

Discover the power of XPaths for web crawling and data extraction in this expert guide. Learn how to write effective XPaths with real-world examples.

What Can I Collect with a Web Crawler?

February 15, 2023 | By admin

The TL-DR This article outlines the types of data that can be collected with a web crawler for company analysis, • Read More »

The Best Methods to Extract Data from AJAX Pages

January 20, 2023 | By admin

Can web crawlers interpret and extract information from JavaScript or AJAX pages? Absolutely, but it requires using a system that • Read More »

Is Web Crawling Legal?

January 8, 2023 | By admin

While web crawlers can be useful and even necessary in some cases, using them can also raise significant legal concerns.

Web Crawlers and Proxies: How to Use Proxies with PHP Web Crawlers

Shield Your IP Address: Web Crawlers and Proxies

July 16, 2019 | By admin

Tired of your web crawlers getting blocked? Try using a free proxy. In this article we explain what proxies are, how to use them, and where to get them.

Web Crawler Development Techniques

Web Crawler Development Techniques

July 15, 2019 | By admin

Looking for some quick code to build your web crawler in PHP? Here's some code we use a lot here at Potent Pages to make our development a lot easier!

How to create a polite PHP web crawler using robot.txt.

Creating a Polite PHP Web Crawler: Checking robots.txt

May 31, 2018 | By admin

In this tutorial, we create a PHP website spider that uses the robots.txt file to know which pages we're allowed to download. We continue from our previous tutorials to create a robust web spider and expand on it to check for download crawling permissions.

How to create a simple PHP web crawler to download a website

Creating a Simple PHP Web Crawler

May 24, 2018 | By admin

Looking to download a site or multiple webpages? Interested in examining all of the titles and descriptions for a site? We created a quick tutorial on building a script to do this in PHP. Learn how to download webpages and follow links to download an entire website.

Scroll To Top