Website Rendering Instructions

This endpoint costs 2 credits per request. Contact us if you need help or have any questions.

If you need to scrape a website, Piloterr’s Website Crawler API is the perfect solution. It’s a high-performance API that can scrape any website and return the HTML content. You can use it to scrape any website, including those with JavaScript rendering, and bypass Cloudflare, Akamai, PerimeterX, and DataDome. The API is easy to use and requires only a website URL and an API key. You can also set parameters to control the API’s behavior, such as following redirects, waiting for JavaScript to render, and waiting for specific elements to appear in the DOM. The API is perfect for web scraping, data extraction, and content analysis.

For security and ethical reasons, Piloterr implements a comprehensive filtering system for websites accessible through this endpoint. This measure helps prevent misuse and ensures compliance with legal and ethical standards. If you wish to use this endpoint, please contact our support team to request the addition of your domain to our allowlist. We’ll review each request carefully, considering factors such as content appropriateness, legal compliance, and potential impact on our infrastructure. This process helps maintain the quality and reliability of our service for all users.

x-api-key

string

required

This parameter specifies the private key you’ll need for Piloterr access.

Parameters

query

string

A website URL with either the http or https protocol.

wait_in_seconds

integer

Some code-heavy websites need time to fully “render”. To direct Piloterr to wait before it returns the fully rendered HTML, use the wait parameter with a value in seconds between 0 and 30. The Piloterr headless browsers will then wait the duration of the time set in seconds before returning the page’s HTML.

wait_for

string

It’s sometimes necessary to wait for a particular element to appear in the DOM before Piloterr returns the HTML content. Our headless browsers will wait for the CSS / Xpath selector passed in the parameter before returning the HTML. For example:

To wait for an element with ID: wait_for=#loading-container
To wait for an element with class: wait_for=.content-loaded
To wait for a specific element: wait_for=div.main-content#user-profile

block_ads

boolean

By default, Piloterr does not block ads. Set this parameter to true to block ads.

timeout

integer

Set this parameter to the maximum number of seconds Piloterr will wait for the page to load.

browser_instructions

array

An array of browser navigation instructions to execute during page rendering. This allows you to control scrolling and other browser actions to trigger dynamic content loading or simulate human behavior.Currently supports two instruction types:

scroll
scroll_to_bottom

Please refer to the Browser Instructions section for more details.

Browser Instructions

This functionality allows executing navigation instructions in the browser to control scrolling and other actions.

Supported Instruction Types

1. Scroll by pixels (`scroll`)

Allows scrolling the page by a precise number of pixels. Parameters:

type: "scroll"
x: Number of pixels to scroll horizontally (positive = right, negative = left)
y: Number of pixels to scroll vertically (positive = down, negative = up)
duration: (optional) Duration in seconds for smooth scrolling (default: 0 = instant)
wait_time_s: (optional) Wait time in seconds after the instruction (default: 0)

Example:

{
    "type": "scroll",
    "x": 0,
    "y": 4115,
    "duration": 5,
    "wait_time_s": 2
}

2. Scroll to bottom (`scroll_to_bottom`)

Automatically scrolls to the bottom of the page. Parameters:

type: "scroll_to_bottom"
duration: (optional) Duration in seconds for smooth scrolling (default: 0 = instant)
wait_time_s: (optional) Wait time in seconds after the instruction (default: 0)

Example:

{
    "type": "scroll_to_bottom",
    "duration": 3,
    "wait_time_s": 2
}

Usage

Add the browser_instructions parameter to your request with a list of instructions:

Example 1: Instant scroll

{
    "query": "https://www.comparably.com/companies/airbus",
    "wait_for": "#ng-state",
    "browser_instructions": [
        {
            "type": "scroll",
            "x": 0,
            "y": 4115,
            "wait_time_s": 5
        }
    ]
}

Example 2: Smooth scroll over 5 seconds

{
    "query": "https://www.comparably.com/companies/airbus",
    "wait_for": "#ng-state",
    "browser_instructions": [
        {
            "type": "scroll",
            "x": 0,
            "y": 4115,
            "duration": 5,
            "wait_time_s": 2
        }
    ]
}

Example 3: Scroll to bottom with duration

{
    "query": "https://www.comparably.com/companies/airbus",
    "wait_for": "#ng-state",
    "browser_instructions": [
        {
            "type": "scroll_to_bottom",
            "duration": 10,
            "wait_time_s": 1
        }
    ]
}

Example 4: Multiple instructions

{
    "query": "https://www.comparably.com/companies/airbus",
    "wait_for": "#ng-state",
    "browser_instructions": [
        {
            "type": "scroll",
            "x": 0,
            "y": 1000,
            "duration": 2,
            "wait_time_s": 1
        },
        {
            "type": "scroll",
            "x": 0,
            "y": 2000,
            "duration": 3,
            "wait_time_s": 1
        },
        {
            "type": "scroll_to_bottom",
            "duration": 4,
            "wait_time_s": 2
        }
    ]
}

Duration Parameter

The duration parameter controls the smoothness of scrolling:

duration: 0 (default): Instant scroll, immediate movement to final position
duration: > 0: Progressive scroll over the specified duration
- The scroll is divided into small steps (minimum 10, or 5 per second)
- Each step waits a short delay to create a smooth effect
- More natural for simulating human behavior

Use Cases

Triggering lazy loading

{
    "type": "scroll",
    "x": 0,
    "y": 4115,
    "duration": 3,
    "wait_time_s": 5
}

Loading dynamic content

{
    "type": "scroll_to_bottom",
    "duration": 2,
    "wait_time_s": 3
}

[
    {
        "type": "scroll",
        "x": 0,
        "y": 1000,
        "duration": 2,
        "wait_time_s": 1
    },
    {
        "type": "scroll",
        "x": 0,
        "y": 2000,
        "duration": 2,
        "wait_time_s": 1
    }
]

Important Notes

Execution order: Instructions are executed sequentially in the specified order
Error handling: If an instruction fails, subsequent ones continue to execute
Timeout: Increase timeout if using significant durations
Performance: Smooth scrolling (duration > 0) takes more time but better simulates human behavior

Response

<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
        body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont,
        "Segoe UI",
        "Open Sans",
        "Helvetica Neue", Helvetica, Arial, sans-serif;
    }
        div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,
        0,
        0,
        0.02);
    }
        a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
        @media (max-width: 700px) {
        div {
        margin: 0 auto;
        width: auto;
    }
    }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
        domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

Website API

Website Rendering Instructions

Header

Parameters

Browser Instructions

Supported Instruction Types

1. Scroll by pixels (`scroll`)

2. Scroll to bottom (`scroll_to_bottom`)

Usage

Example 1: Instant scroll

Example 2: Smooth scroll over 5 seconds

Example 3: Scroll to bottom with duration

Example 4: Multiple instructions

Duration Parameter

Use Cases

Triggering lazy loading

Loading dynamic content

Smooth navigation to avoid detection

Important Notes

Response

Website API

​Header

​Parameters

​Browser Instructions

​Supported Instruction Types

​1. Scroll by pixels (scroll)

​2. Scroll to bottom (scroll_to_bottom)

​Usage

​Example 1: Instant scroll

​Example 2: Smooth scroll over 5 seconds

​Example 3: Scroll to bottom with duration

​Example 4: Multiple instructions

​Duration Parameter

​Use Cases

​Triggering lazy loading

​Loading dynamic content

​Smooth navigation to avoid detection

​Important Notes

​Response

Header

Parameters

Browser Instructions

Supported Instruction Types

1. Scroll by pixels (`scroll`)

2. Scroll to bottom (`scroll_to_bottom`)

Usage

Example 1: Instant scroll

Example 2: Smooth scroll over 5 seconds

Example 3: Scroll to bottom with duration

Example 4: Multiple instructions

Duration Parameter

Use Cases

Triggering lazy loading

Loading dynamic content

Smooth navigation to avoid detection

Important Notes

Response