Piloterr

Website Crawler

A robust solution for efficiently extracting a wide range of data from web pages via high-performance request-mode crawling.

Credit cost: 1 credit per call

GET
/v2/website/crawler

Authorization

ApiKeyAuth
x-api-key<token>

Your API key. Include it in the x-api-key header for every request.

In: header

Query Parameters

query*string

A website URL to crawl. Must include the http:// or https:// protocol.

Example:

https://example.com

Notes:

  • Only public pages are accessible. Pages behind authentication will not return useful content.
  • For pages requiring JavaScript rendering, use the Website Rendering or Website WebUnlocker endpoint instead.
  • Choosing the right API can be tricky this guide will point you to the best option: Website Scraping
allow_redirects?boolean

Controls whether the crawler should follow HTTP redirects.

  • true: Follow redirects automatically (default)
  • false: Stop at the first HTTP response without following any redirect

Notes:

  • Set to false when you need to inspect the redirect chain itself (e.g. detecting 301 vs 302 codes).
Default"true"
return_page_source?boolean

Controls whether the response returns the raw, unprocessed HTML source of the page.

  • false: Return processed/parsed content (default)
  • true: Return the exact HTML source as delivered by the web server

Notes:

  • Useful for debugging or when you need the precise markup before any transformation.
Default"false"

Response Body

application/json

application/json

application/json

application/json

application/json

application/json

curl -X GET "https://api.piloterr.com/v2/website/crawler?query=string"

"<!doctype html><html lang=\"en\"><head><title>Example Domain</title><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}h1{font-size:1.5em}div{opacity:0.8}a:link,a:visited{color:#348}</style></head><body><div><h1>Example Domain</h1><p>This domain is for use in documentation examples without needing permission. Avoid use in operations.</p><p><a href=\"https://iana.org/domains/example\">Learn more</a></p></div></body></html>"

{
  "error": "Bad Request"
}
{
  "error": "Invalid API Key"
}
{
  "error": "Payment required"
}
{
  "error": "Rate limit exceeded for the API key: quota monthly"
}
{
  "error": "Internal Error"
}