Crawl Any Website
Retrieve the raw HTML of any webpage via the Piloterr API and extract structured data without maintaining your own headless browser infrastructure.
Overview
This playbook shows how to fetch the full HTML of any webpage using the Piloterr Website Crawler, then extract specific data from it. A typical use case is competitor price monitoring: crawl a product page daily and parse the price from the HTML.
Prerequisites
- A Piloterr API key — get one at app.piloterr.com
- Install dependencies for your language:
pip install requests beautifulsoup4Steps
Crawl a webpage
Call GET /v2/website/crawler with the query parameter set to the target URL. The response is the raw HTML string (JSON-encoded).
import requests
API_KEY = "YOUR_API_KEY"
response = requests.get(
"https://api.piloterr.com/v2/website/crawler",
headers={"x-api-key": API_KEY},
params={"query": "https://example.com", "allow_redirects": "true"},
)
html = response.json() # returns the HTML as a string
print(html[:500])Extract data from the HTML
Parse the HTML to extract specific elements — here we extract the page title and all <h1> headings.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
title = soup.find("title")
print("Page title:", title.text if title else "N/A")
for h in soup.find_all("h1"):
print("H1:", h.get_text(strip=True))Build a price monitoring script
Crawl a product page and extract the price using a CSS selector.
import requests
from bs4 import BeautifulSoup
API_KEY = "YOUR_API_KEY"
WATCH_URL = "https://www.example-shop.com/product/123"
def crawl(url: str) -> str:
r = requests.get("https://api.piloterr.com/v2/website/crawler",
headers={"x-api-key": API_KEY}, params={"query": url, "allow_redirects": "true"})
return r.json()
def extract_price(html: str) -> str | None:
soup = BeautifulSoup(html, "html.parser")
el = soup.select_one("[data-price], .price, #price")
return el.get_text(strip=True) if el else None
price = extract_price(crawl(WATCH_URL))
print(f"Current price: {price}" if price else "Price element not found.")Set allow_redirects=true to follow HTTP 301/302 redirects automatically — useful for short URLs or e-commerce platforms that redirect product pages.