Archives: web-crawler-development
The Best Methods to Extract Data from AJAX Pages
January 20, 2023 | By David Selden-TreimanCan web crawlers interpret and extract information from JavaScript or AJAX pages? Absolutely, but it requires using a system that • Read More »
Is Web Crawling Legal?
January 8, 2023 | By David Selden-TreimanWhile web crawlers can be useful and even necessary in some cases, using them can also raise significant legal concerns.
Shield Your IP Address: Web Crawlers and Proxies
July 16, 2019 | By David Selden-TreimanTired of your web crawlers getting blocked? Try using a free proxy. In this article we explain what proxies are, how to use them, and where to get them.
Web Crawler Development Techniques
July 15, 2019 | By David Selden-TreimanLooking for some quick code to build your web crawler in PHP? Here's some code we use a lot here at Potent Pages to make our development a lot easier!
Creating a Polite PHP Web Crawler: Checking robots.txt
May 31, 2018 | By David Selden-TreimanIn this tutorial, we create a PHP website spider that uses the robots.txt file to know which pages we're allowed to download. We continue from our previous tutorials to create a robust web spider and expand on it to check for download crawling permissions.
Creating a Simple PHP Web Crawler
May 24, 2018 | By David Selden-TreimanLooking to download a site or multiple webpages? Interested in examining all of the titles and descriptions for a site? We created a quick tutorial on building a script to do this in PHP. Learn how to download webpages and follow links to download an entire website.
Downloading a Webpage using PHP and cURL
May 24, 2018 | By David Selden-TreimanLooking to automatically download webpages? Here's how to download a page using PHP and cURL.