Piloterr

Website Crawler

Fetch public pages with fast HTTP crawling (no JavaScript). Best first choice for static or server-rendered HTML; use Rendering or WebUnlocker when JS or anti-bot blocks you.

Credit cost: 1 credit per call

GET
/v2/website/crawler

Authorization

ApiKeyAuth
x-api-key<token>

Your API key. Include it in the x-api-key header for every request.

In: header

Query Parameters

query*string

A public website URL to fetch with the Crawler engine (HTTP only, no JavaScript). Must include http:// or https://.

Example:

https://example.com

Choosing the right engine:

See Website Scraping guide for a full comparison.

allow_redirects?boolean

Controls whether the crawler should follow HTTP redirects.

  • true: Follow redirects automatically (default)
  • false: Stop at the first HTTP response without following any redirect

Notes:

  • Set to false when you need to inspect the redirect chain itself (e.g. detecting 301 vs 302 codes).
Default"true"
return_page_source?boolean

Controls whether the response returns the raw, unprocessed HTML source of the page.

  • false: Return processed/parsed content (default)
  • true: Return the exact HTML source as delivered by the web server

Notes:

  • Useful for debugging or when you need the precise markup before any transformation.
Default"false"

Response Body

application/json

application/json

application/json

application/json

application/json

application/json

curl -X GET "https://api.piloterr.com/v2/website/crawler?query=string"

"<!doctype html><html lang=\"en\"><head><title>Example Domain</title><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}h1{font-size:1.5em}div{opacity:0.8}a:link,a:visited{color:#348}</style></head><body><div><h1>Example Domain</h1><p>This domain is for use in documentation examples without needing permission. Avoid use in operations.</p><p><a href=\"https://iana.org/domains/example\">Learn more</a></p></div></body></html>"

{
  "error": "Bad Request"
}
{
  "error": "Invalid API Key"
}
{
  "error": "Payment required"
}
{
  "error": "Rate limit exceeded for the API key: quota monthly"
}
{
  "error": "Internal Error"
}