Fetch opens each page in a real browser and runs its JavaScript before writing the Markdown, State JSON, and a Snapshot you can replay.
Fetch opens every page in a real browser and runs its JavaScript before reading it. That makes the difference on pages where the raw HTML misses what you need:
Fetch decides when the page is ready. There's no option to set a timeout, a wait selector, network idle, or other readiness rules.
What to expect:
Fetchingโฆ
Waiting for a browserโฆ
Fetchingโฆ
Waiting for a browserโฆ
Fetchingโฆ
Waiting for a browserโฆ
Every Fetch that waits for its result has one 150-second budget. It starts when the request is accepted and covers everything after that: queueing, loading, settling, scrolling, the Snapshot, and delivery.
504 FetchCaptureTimeout. The TypeScript SDK doesn't retry it, so an outage on our side isn't amplified. A manual retry can still help. The timeout doesn't prove the page was at fault.529 FetchCapacityTimeout with Retry-After: 5. It's a capacity problem, not a page problem. The TypeScript SDK retries it.timeoutMs defaults to 60 seconds and cuts off long captures, so raise it. See TypeScript SDK.504, timeoutMs reports the ceiling, 150000 by default. The timings inside the budget, such as readiness waits, are internal and can change.scrollFullPage is the one public browser behavior control. It defaults to false. When true, Fetch attempts to scroll the page before capture to trigger lazy-loaded content, then returns to the top before capturing.
Use scrollFullPage when important content appears only after scrolling.
import { ExpandClient } from '@expandai/sdk'
const expand = new ExpandClient()
const markdown = await expand.fetch({
url: 'https://stripe.com/pricing',
browserConfig: { scrollFullPage: true },
})curl https://api.expand.ai/v1/fetch \
-H "x-expand-api-key: $EXPAND_API_KEY" \
-H
Fetch can take a full-page or viewport screenshot of any page it opens in a browser. To get one, set include.screenshot in the request body, not browserConfig. It comes back in the JSON object as data.screenshot, with its url and a fullPage flag. Anyone with the url can open the image, and it expires 15 minutes after the response; fetch again for a fresh one. Markdown Mode can't include the screenshot, but it's still saved with the Snapshot. JSON Mode returns a link to it. See Two modes.
import { ExpandClient } from '@expandai/sdk'
const expand = new ExpandClient()
const page = await expand.fetchJson({
url: 'https://example.com',
include: { screenshot: true },
})curl https://api.expand.ai/v1/fetch/json \
-H "x-expand-api-key: $EXPAND_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"include": { "screenshot": true }
}'The full schema is in the API reference. Everything else you can include, and how each mode returns it, is on Fetching a URL.
Fetch follows redirects and captures the page it lands on. Each mode tells you where it landed in its own way:
{
"meta": {
"url": "https://www.example.com/" // where Fetch landed
}
}---
url: http://example.com # what you asked for
finalUrl: https://www.example.com/ # where Fetch landed, only after a redirect
---data.response.Fetch uses a real browser, so it reads many sites that block plain HTTP requests. When a site puts a bot check or CAPTCHA in front of a public page, Fetch gets past common ones automatically, such as "Just a moment..." checks, Cloudflare challenges and Turnstile, reCAPTCHA, and checkbox challenges. Fetch, Batched Fetch, and Crawl all do this, and you're charged once for the page.
Fetch reads public pages only. It doesn't sign in, pass paywalls, or open content behind other access controls. Some sites still refuse it, and those requests end as FetchBlocked.
A blocked page returns 403 FetchBlocked. Its blockedType says why: captcha, botProtection, ipBlock, or authWall. You are not charged for blocked requests.
{
"_tag": "FetchBlocked",
"url": "https://example.com",
"blockedType"
A capture that was cut short can still return content. In the JSON object, meta.blocked or meta.loadingStatus marks it as partial. In a Batched Fetch, each item reports its own blocked state.
You can set three browser options. Fetch manages everything else.
| Option | Field | Definition |
|---|---|---|
| Scrolling | scrollFullPage | Best-effort scroll before capture. See Scrolling. |
| Screenshot | include.screenshot | Captures a screenshot. See Screenshots. |
| Screenshot size |
These are managed by Fetch and aren't options today:
| Control | Definition |
|---|---|
| Timeout | Readiness is managed internally. Synchronous Fetch enforces one 150 s budget per request. Batched items get their own when scheduled. |
| Viewport / device | Not a public Fetch option. |
| User agent | Not a public Fetch option. |
| Cookies / sign-in | Not a public Fetch option. You can't send cookies or sign in. |
If any of these become public, this page and the API reference change together.
When output looks incomplete or surprising, open the replay. It shows what Fetch saw at capture time.
Look for content that loaded late, a capture that was cut short, or an unexpected redirect. Playground & replay covers the full workflow.
When a result looks wrong, open the replay before assuming the page is unreachable. See Debug with replay.
include.screenshot.fullPage |
Defaults to full page if not set. Set to false for the viewport only. |
| Geolocation |
| Not a public Fetch option. |
| Wait selector / action steps | Not a public Fetch option. Fetch reads pages and doesn't drive them. |
| Bot checks / CAPTCHAs | Not a public Fetch option. Fetch handles them on its own. See Blocked pages. |