Archives: Web Crawler
The Best Methods to Extract Data from AJAX Pages
January 20, 2023 | By adminCan web crawlers interpret and extract information from JavaScript or AJAX pages? Absolutely, but it requires using a system that • Read More »
Is Web Crawling Legal?
January 8, 2023 | By adminWhile web crawlers can be useful and even necessary for certain legitimate purposes, their use can also raise significant legal concerns, particularly when it comes to issues of privacy, copyright, and computer fraud. Make sure to use them legally and ethically.
Shield Your IP Address: Web Crawlers and Proxies
July 16, 2019 | By adminTired of your web crawlers getting blocked? Try using a free proxy. In this article we explain what proxies are, how to use them, and where to get them.
Web Crawler Development Techniques
July 15, 2019 | By adminLooking for some quick code to build your web crawler in PHP? Here's some code we use a lot here at Potent Pages to make our development a lot easier!
Creating a Polite PHP Web Crawler: Checking robots.txt
May 31, 2018 | By adminIn this tutorial, we create a PHP website spider that uses the robots.txt file to know which pages we're allowed to download. We continue from our previous tutorials to create a robust web spider and expand on it to check for download crawling permissions.
Creating a Simple PHP Web Crawler
May 24, 2018 | By adminLooking to download a site or multiple webpages? Interested in examining all of the titles and descriptions for a site? We created a quick tutorial on building a script to do this in PHP. Learn how to download webpages and follow links to download an entire website.
Downloading a Webpage using PHP and cURL
May 24, 2018 | By adminLooking to automatically download webpages? Here's how to download a page using PHP and cURL.