> ## Documentation Index
> Fetch the complete documentation index at: https://doc.agentfirst.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Browsing

> Unblock difficult-to-crawl websites.

export const Autoscaling = () => {
  return <Warning>
      This endpoint autoscales; if you receive a <code>503</code> response, maintain traffic and
      expect errors to resolve within a minute.
    </Warning>;
};

export const Reference = ({endpoint}) => {
  return <Info>
      See also the <Link href={`reference/${endpoint}`}>API reference</Link> for more detail.
    </Info>;
};

export const apiEndpoint = 'api.agentfirst.dev';

export const emailAddress = 'brain@agentfirst.dev';

export const companyName = 'Agent First';

<Reference endpoint="browser" />

## Service usage

**{companyName}'s** browser service navigates intelligently on your behalf, rendering JavaScript,
solving [captchas](https://www.ibm.com/topics/captcha), and retrying failed requests. The service
has a RESTful interface that accepts `GET` requests at
**https\://‍{apiEndpoint}/browser** and a
[Chrome DevTools Protocol (CDP) interface](#cdp-interaction) for controlling browser navigation
yourself that accepts WebSocket connections at **wss\://{apiEndpoint}/browser**.

Webpages can be localized with <Link href="geotargeting">geotargeting parameters</Link>. Those
requested via REST are retrieved in real time by default or optionally queued for <Link href="scheduling">subsequent retrieval</Link>. Up to **3 minutes** is allotted per real-time API
call to accommodate captcha-solving, multiple retries, and other mitigations. <Link href="usage">Usage data</Link> is published regularly.

<Tip>
  If a site you're targeting remains blocked, even with different browsing params, <a href={`mailto:${emailAddress}`}>let us know</a>. We can usually help unblock any site within **48
  hours**.
</Tip>

### Authentication

You can access the service by including your secret API token in an `Authorization` header:

```http theme={null}
Authorization: Bearer [API token here]
```

Here's an example request that uses the [common Curl command](https://curl.se/):

```shell theme={null}
curl -H "Authorization: Bearer $AGENT_FIRST_TOKEN" \
'https://api.agentfirst.dev/browser'\
'?url=https://example.com/'
```

<Autoscaling />

### REST parameters

Besides the [geotargeting and scheduling params linked above](#service-usage), required and optional
REST params can be added in a
[standard query string](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL#parameters).
See <Link href="pricing">our rate card</Link> for the prices of premium params. The keys and values
**{companyName}** supports are as follows:

| Key          | Required | Premium | Value                                                                                                                                                                                                                                                                                               |
| :----------- | :------- | :------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`        | ✅        | ⬜       | The URL of up to 2,047 characters of the page to browse; any unsafe characters require [URL encoding](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding)                                                                                                                           |
| `difficulty` | ⬜        | ✅       | The difficulty pool to attempt to access the URL from, `low`, `medium`, or `high` (planned); `low` is the default difficulty                                                                                                                                                                        |
| `speed`      | ⬜        | ✅       | The speed to attempt to access the URL at, `light`, `ridiculous`, or `ludicrous` (planned), where `ridiculous` is 30 percent faster on average than `light` speed; `light` is the default speed                                                                                                     |
| `device`     | ⬜        | ⬜       | The name as returned by the [devices resource](#device-emulation) of the device to emulate browsing on (these names are case insensitive but must include form- or URL-encoded spaces and punctuation marks); device emulation is unused by default                                                 |
| `session`    | ⬜        | ⬜       | Any unique identifier of up to 255 characters (regardless of character encoding); **{companyName}** will make best efforts to route calls in the same session to the same egress node for up to 12 minutes                                                                                          |
| `captcha`    | ⬜        | ⬜       | The intended resolution of any detected captcha, `solved`, `ignored`, or `rejected`, where `rejected` results in a `403` response; `solved` is the default captcha resolution                                                                                                                       |
| `readiness`  | ⬜        | ⬜       | The standard ready event to await before snapshotting browsed content, `load` or `domcontentloaded`; `load` is the default ready event                                                                                                                                                              |
| `delay`      | ⬜        | ⬜       | The number of supplemental seconds to delay before snapshotting browsed content, from `.1` to `10` inclusive; no delay is used by default                                                                                                                                                           |
| `format`     | ⬜        | ⬜       | The format to output to, `rendered`, `raw`, or `markdown` (see the [section below](#response-format)); `rendered` is the default format                                                                                                                                                             |
| `expiration` | ⬜        | ⬜       | The age in days of when to consider cached content expired, where `0` disables caching; `1` is the default number of days before expiration                                                                                                                                                         |
| `callback`   | ⬜        | ⬜       | The encoded HTTP or HTTPS callback URL or Amazon SQS queue URL or ARN of up to 2,047 characters to notify when the (prerequisite) asynchronous content has been retrieved; any SQS queue must grant `sqs:SendMessage` permission to the `arn:aws:iam::180363035301:role/api-instance` AWS principal |

### CDP interaction

If your use case involves interacting with not just reading pages, you can connect
[Puppeteer](https://pptr.dev/), [Playwright](https://playwright.dev/), or
[another CDP-compatible automation framework](https://github.com/ChromeDevTools/awesome-chrome-devtools#libraries-for-driving-the-protocol-or-a-layer-above)
to the browser service:

```js theme={null}
#!/usr/bin/env node
import puppeteer from 'puppeteer';

const browser = await puppeteer.connect({
  browserWSEndpoint: 'wss://api.agentfirst.dev/browser',
  headers: {
    Authorization: `Bearer ${process.env.AGENT_FIRST_TOKEN}`
  }
});
```

<Accordion icon="terminal" title="View Playwright sample">
  ```py theme={null}
  #!/usr/bin/env python3
  import os
  from playwright.sync_api import sync_playwright

  with sync_playwright() as playwright:
      browser = playwright.chromium.connect_over_cdp(
          'wss://api.agentfirst.dev/browser',
          headers={
              'Authorization':
                  f"Bearer {os.environ['AGENT_FIRST_TOKEN']}"
          }
      )
  ```
</Accordion>

The connection string takes geotargeting params and a subset of the [REST params](#rest-parameters),
namely `difficulty`, `device`, and `session`:

```js theme={null}
const browser = await puppeteer.connect({
  browserWSEndpoint:
    'wss://api.agentfirst.dev/browser' +
    '?difficulty=medium',
  headers: {
    Authorization: `Bearer ${process.env.AGENT_FIRST_TOKEN}`
  }
});
```

After connecting, most of your framework's API (see the [Puppeteer reference](https://pptr.dev/api),
[Playwright reference](https://playwright.dev/docs/api/class-playwright), et cetera) will be
available:

```js theme={null}
const [page] = await browser.pages();

await page.goto('https://example.com/');
console.log('URL:', page.url());
console.log('Title:', await page.title());
```

CDP use is metered by time, so you should disconnect from the service when done by calling
`browser.disconnect()` (Puppeteer), `browser.close()` (Playwright), or the equivalent.

### Protocol mixture

You can control costs by mixing REST requests and CDP use within a sticky browser session. E.g., you
may want to prepare a page for interaction over REST then interact with the page over CDP:

<Accordion icon="terminal" title="View REST-to-CDP sample">
  ```js theme={null}
  #!/usr/bin/env node
  import puppeteer from 'puppeteer';

  const restEndpoint = 'https://api.agentfirst.dev/browser';
  const cdpEndpoint = 'wss://api.agentfirst.dev/browser';
  const apiToken = process.env.AGENT_FIRST_TOKEN;
  const headers = { Authorization: `Bearer ${apiToken}` };
  const sessionId = Date.now();
  const orderUrl = 'https://httpbin.org/forms/post';
  const sizeSelector = '[name="size"][value="small"]';
  const toppingSelector = '[name="topping"][value="cheese"]';
  const buttonSelector = 'button';
  const requestTimeoutMs = 180_000;
  const connectionTimeoutMs = 10_000;
  const delayMs = 1_000;
  const delay = (ms) => {
    return new Promise((resolve) => {
      setTimeout(resolve, ms);
    });
  };
  const getCookies = (response) => {
    const cookies = response.headers.getSetCookie?.() ?? [];
    const keyvals = cookies.map((cookie) => {
      return cookie.split(';', 1)[0];
    });

    return keyvals.join('; ');
  };

  console.log(
    'Making REST request with session',
    sessionId,
    '...'
  );

  const controller = new AbortController();
  let timeoutId = setTimeout(() => {
    controller.abort();
  }, requestTimeoutMs);
  let response;

  try {
    const url =
      restEndpoint +
      '?url=' +
      encodeURIComponent(orderUrl) +
      '&session=' +
      sessionId +
      '&expiration=0';
    response = await fetch(
      url,
      { headers, signal: controller.signal }
    );
  } catch (error) {
    const hasTimedOut = error.name == 'AbortError';
    response = new Response(
      hasTimedOut
        ? 'Request timed out'
        : 'Unknown error occurred',
      { status: hasTimedOut ? 504 : 500 }
    );
  } finally {
    clearTimeout(timeoutId);
  }

  if (response.ok) {
    console.log('Order form read successfully');
  } else {
    const text = await response.text();

    console.error('Order form read failure:', text.trimEnd());

    process.exit(1);
  }

  console.log(
    'Connecting via CDP with session',
    sessionId,
    '...'
  );

  timeoutId = setTimeout(() => {
    console.error('Connection timed out');

    process.exit(1);
  }, connectionTimeoutMs);
  const url = `${cdpEndpoint}?session=${sessionId}`;
  const browser = await puppeteer.connect({
    browserWSEndpoint: url,
    headers: { ...headers, Cookie: getCookies(response) }
  });

  clearTimeout(timeoutId);

  const page = (await browser.pages()).find((page) => {
    return page.url().includes(orderUrl);
  });

  if (page) {
    console.log('Session handed off successfully');
  } else {
    console.error('Session handoff failure');

    process.exit(1);
  }

  await page.waitForSelector(sizeSelector);
  await page.click(sizeSelector);
  await delay(delayMs);
  await page.waitForSelector(toppingSelector);
  await page.click(toppingSelector);
  await delay(delayMs);
  await page.waitForSelector(buttonSelector);
  await Promise.all([
    page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
    page.click(buttonSelector)
  ]);

  try {
    const json = await page.evaluate(() => {
      return document.body.innerText;
    });
    const data = JSON.parse(json);

    if (
      data.form &&
      typeof data.form.size == 'string' &&
      typeof data.form.topping == 'string'
    ) {
      console.log(
        'Ordered',
        data.form.size.toLowerCase(),
        'pizza with',
        data.form.topping.toLowerCase()
      );
    } else {
      console.error('Receipt was malformed');

      process.exit(1);
    }
  } catch (error) {
    console.error('Receipt parse failure:', error.message);

    process.exit(1);
  }

  await browser.disconnect();
  ```
</Accordion>

To hand off in reverse, from CDP to REST, call your disconnect method first.

### Device emulation

The `device` param lets you browse device-specific content, rather than the default desktop content.
For a list of supported smartphone and tablet devices, make a request with your API token and no
params to **https\://‍{apiEndpoint}/browser/devices**:

```shell theme={null}
curl -H "Authorization: Bearer $AGENT_FIRST_TOKEN" \
'https://api.agentfirst.dev/browser/devices'
```

The API will return JSON that contains an alphabetized array of device names:

```json theme={null}
[
  "Blackberry PlayBook",
  "Blackberry PlayBook landscape",
  "BlackBerry Z30",
  "[Remaining device names here]"
]
```

Then, pass your device name of choice into your browsing request or connection:

```shell theme={null}
curl -H "Authorization: Bearer $AGENT_FIRST_TOKEN" \
'https://api.agentfirst.dev/browser'\
'?url=https://crackberry.com/'\
'&device=Blackberry+Playbook'
```

```js theme={null}
const browser = await puppeteer.connect({
  browserWSEndpoint:
    'wss://api.agentfirst.dev/browser' +
    '?device=Blackberry+Playbook',
  headers: {
    Authorization: `Bearer ${process.env.AGENT_FIRST_TOKEN}`
  }
});
```

### Response format

Read-only pages are returned as rendered HTML, raw (unrendered) HTML, or Markdown optimized for LLM
prompts.

## Additional examples

### Cachebusting

```shell theme={null}
curl -H "Authorization: Bearer $AGENT_FIRST_TOKEN" \
'https://api.agentfirst.dev/browser'\
'?url=https://api.weather.gov/alerts'\
'&expiration=0'
```

### End-to-end Playwright

<Accordion icon="terminal" title="View end-to-end sample">
  ```py theme={null}
  #!/usr/bin/env python3
  import os
  from playwright.sync_api import sync_playwright

  ENDPOINT = 'wss://api.agentfirst.dev/browser'
  API_TOKEN = os.environ['AGENT_FIRST_TOKEN']
  URL = 'https://example.com/'

  print(f"Connecting to {ENDPOINT} ...")

  with sync_playwright() as playwright:
      browser = playwright.chromium.connect_over_cdp(
          ENDPOINT,
          headers={'Authorization': f"Bearer {API_TOKEN}"}
      )
      context = browser.contexts[0]
      page = context.pages[0]

      print(f"Going to {URL} ...")

      page.goto(URL)
      print(f"URL: {page.url}")
      print(f"Title: {page.title()}")
      browser.close()
  ```
</Accordion>