Give us a call: (800) 252-6164
Select your language

Archives: web-crawler-development

The Best Methods to Extract Data from AJAX Pages

January 20, 2023 | By admin

Can web crawlers interpret and extract information from JavaScript or AJAX pages? Absolutely, but it requires using a system that • Read More »

Is Web Crawling Legal?

January 8, 2023 | By admin

While web crawlers can be useful and even necessary for certain legitimate purposes, their use can also raise significant legal concerns, particularly when it comes to issues of privacy, copyright, and computer fraud. Make sure to use them legally and ethically.

Web Crawlers and Proxies: How to Use Proxies with PHP Web Crawlers

Shield Your IP Address: Web Crawlers and Proxies

July 16, 2019 | By admin

Tired of your web crawlers getting blocked? Try using a free proxy. In this article we explain what proxies are, how to use them, and where to get them.

Web Crawler Development Techniques

Web Crawler Development Techniques

July 15, 2019 | By admin

Looking for some quick code to build your web crawler in PHP? Here's some code we use a lot here at Potent Pages to make our development a lot easier!

How to create a polite PHP web crawler using robot.txt.

Creating a Polite PHP Web Crawler: Checking robots.txt

May 31, 2018 | By admin

In this tutorial, we create a PHP website spider that uses the robots.txt file to know which pages we're allowed to download. We continue from our previous tutorials to create a robust web spider and expand on it to check for download crawling permissions.

How to create a simple PHP web crawler to download a website

Creating a Simple PHP Web Crawler

May 24, 2018 | By admin

Looking to download a site or multiple webpages? Interested in examining all of the titles and descriptions for a site? We created a quick tutorial on building a script to do this in PHP. Learn how to download webpages and follow links to download an entire website.

How to Download a Webpage using PHP and cURL

Downloading a Webpage using PHP and cURL

May 24, 2018 | By admin

Looking to automatically download webpages? Here's how to download a page using PHP and cURL.

Scroll To Top