Skip to main content
GET
/
api
/
v2
/
website
/
crawler
Website Crawler
curl --request GET \
  --url https://piloterr.com/api/v2/website/crawler \
  --header 'x-api-key: <x-api-key>'
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
        body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont,
        "Segoe UI",
        "Open Sans",
        "Helvetica Neue", Helvetica, Arial, sans-serif;
    }
        div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,
        0,
        0,
        0.02);
    }
        a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
        @media (max-width: 700px) {
        div {
        margin: 0 auto;
        width: auto;
    }
    }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
        domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
If you need to scrape a website, Piloterr’s Website Crawler API is the perfect solution. It’s a high-performance API that can scrape any website and return the HTML content. You can use it to scrape any website. If you are interested in js rendering, please see the Website Rendering API or Website WebUnlocker if you need an advanced solution.
This endpoint costs 1 credit per request. Contact us if you need help or have any questions.
​
x-api-key
string
required
This parameter specifies the private key you’ll need for Piloterr access.

​
Parameters

​
query
string
A website URL with either the http or https protocol.
​
allow_redirects
boolean
If set to false, do not follow redirects. true by default.
​
return_page_source
boolean
If set to true, the response will be the HTML source of the page. false by default.

​
Response

<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
        body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont,
        "Segoe UI",
        "Open Sans",
        "Helvetica Neue", Helvetica, Arial, sans-serif;
    }
        div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,
        0,
        0,
        0.02);
    }
        a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
        @media (max-width: 700px) {
        div {
        margin: 0 auto;
        width: auto;
    }
    }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
        domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>