POLITE PHP WEB CRAWLER
How to Check robots.txt Before Downloading Pages

What you’ll build

By the end of this tutorial you’ll have a simple, repeatable pattern you can use in any PHP crawler:

• Download /robots.txt for a host
• Parse the file using a maintained PHP robots.txt parser library
• Set your crawler User-agent and check each URL path before downloading it
• Throttle requests so your crawler behaves politely

Step 1 — Install the robots.txt parser

We’ll use the t1gor/robots-txt-parser Composer package, which supports common directives like User-agent, Allow, Disallow, and crawl delay directives.

composer require t1gor/robots-txt-parser

Then load Composer’s autoloader near the top of your script:

<?php
require __DIR__ . '/vendor/autoload.php';

Step 3 — Download robots.txt safely

robots.txt is typically located at https://example.com/robots.txt. Your crawler should:

• Fetch it once per host
• Handle 404 (no robots.txt) without crashing
• Store the HTTP status code along with the file body
• Cache it (don’t re-download it on every URL)

function http_get($url, $referer = "", $user_agent = ""){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HTTPGET, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);

    if($referer !== ""){
        curl_setopt($ch, CURLOPT_REFERER, $referer);
    }

    $raw = curl_exec($ch);
    if($raw === false){
        $err = curl_error($ch);
        curl_close($ch);
        return ["status" => 0, "headers" => [], "body" => "", "error" => $err];
    }

    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    $header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
    curl_close($ch);

    $header_text = substr($raw, 0, $header_size);
    $body = substr($raw, $header_size);

    return ["status" => $status, "headers_raw" => $header_text, "body" => $body, "error" => ""];
}

Step 5 — Check every URL before downloading it

The rule is simple: never request a page you already know is disallowed. In your crawl loop, check the URL path (not the full URL string) and skip disallowed pages.

// Example crawl loop sketch:
// $queue contains absolute URLs OR paths (depending on your crawler design)

while(!empty($queue)){
    $url = array_shift($queue);

    // Normalize to a path for robots.txt checks
    $parts = parse_url($url);
    $path = $parts['path'] ?? "/";

    // Polite behavior: skip disallowed paths
    if($parser->isDisallowed($path)){
        continue;
    }

    // If allowed, download the page
    $res = http_get($url, $base, $user_agent);

    // TODO: parse page, extract links, push new URLs into $queue
    // TODO: store results in DB, etc.

    // Throttle requests (see next step)
    sleep(1);
}

A cleaner “final script” you can adapt

Below is a compact example that pulls the pieces together. In a production crawler you’d also: cache robots.txt per host, persist a queue, and add monitoring/logging.

<?php
require __DIR__ . '/vendor/autoload.php';

use t1gor\\RobotsTxtParser\\RobotsTxtParser;

$user_agent = "YourCrawlerName/1.0 (+https://potentpages.com/contact/)";
$default_delay = 1;

function http_get($url, $referer = "", $user_agent = ""){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HTTPGET, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
    if($referer !== "") curl_setopt($ch, CURLOPT_REFERER, $referer);

    $raw = curl_exec($ch);
    if($raw === false){
        $err = curl_error($ch);
        curl_close($ch);
        return ["status" => 0, "body" => "", "error" => $err];
    }

    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    $header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
    curl_close($ch);

    $body = substr($raw, $header_size);
    return ["status" => $status, "body" => $body, "error" => ""];
}

// Seed
$seed_url = "https://example.com/";
$seed_parts = parse_url($seed_url);
$base = $seed_parts['scheme'] . "://" . $seed_parts['host'];

// robots.txt
$robots = http_get($base . "/robots.txt", "", $user_agent);
$parser = new RobotsTxtParser($robots['body']);
$parser->setHttpStatusCode((int)$robots['status']);
$parser->setUserAgent($user_agent);

// Simple queue (replace with DB-backed queue in production)
$queue = [$seed_url];
$seen = [];

while(!empty($queue)){
    $url = array_shift($queue);
    if(isset($seen[$url])) continue;
    $seen[$url] = true;

    $parts = parse_url($url);
    $path = $parts['path'] ?? "/";

    if($parser->isDisallowed($path)){
        continue;
    }

    $res = http_get($url, $base, $user_agent);
    if($res['status'] === 429 || $res['status'] === 503){
        // Basic backoff
        sleep(max(5, $default_delay));
        continue;
    }

    // TODO: Extract links and add to $queue (stay on-host, normalize URLs, etc.)
    // TODO: Store content/metadata

    sleep($default_delay);
}

echo "Done\\n";

FAQ: robots.txt, crawl politeness, and PHP crawlers

These are common questions teams ask when they’re moving from a quick script to a crawler that runs reliably and responsibly.