Expand AI logo
DocsDocs
Glow Active
API Reference
Login

Documentation

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ
Browse docs

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ

Fetching a URL

The two modes, what to include in a Fetch, and the pointers to everything held back.

Fetch has two modes. Markdown Mode returns Markdown text, for agents. JSON Mode returns a JSON object, for applications. Both carry the same layers. The Markdown text puts the metadata in YAML frontmatter at the top, and the JSON object puts each layer in its own field.

Markdown Mode is the default. It gives your agent the Main Markdown, enough to work from without flooding its context, and suits first-pass reading, summarization, extraction, and browsing. Include more layers only when the task needs more context or a different artifact. See Progressive disclosure.

Two modes

The same Fetch in both modes, with every layer included.

Markdown textPOST /v1/fetch
Metadata
---
url: news.ycombinator.com
title: Hacker News
playground: expand.land/s/...
---
Main Markdown
# Hacker News

1. Example story title
2. Another story title
Appendix
## Appendix

- [new](/newest) · [past](/front)
State JSON
## Expand State JSON

```json
{ "stories": [ ... ] }
```
MetadataMain MarkdownAppendixState JSON
JSON objectPOST /v1/fetch/json
Metadata
"meta": {
  "url": "news.ycombinator.com",
  "snapshotId": "...",
  "appendix": { "url": "..." }
},
Main Markdown
"markdown":
  "# Hacker News\n\n1. Example…
Appendix

Pick Markdown Mode when the response goes straight to an agent or a prompt. The Markdown text is one string, ready to read.

Pick JSON Mode when your code reads the fields separately, such as the snapshotId or the State JSON.

Note
Links and images on the page's own site come back relative, such as /pricing, or #faq for the page itself. Resolve them against finalUrl, or url when there is no finalUrl. Links to other sites stay absolute.

The Main Markdown always comes back whole, however long the page. Only a Highlights request returns less. Citations point back to the page through evidence IDs: json[].id on State JSON entries, and location.evidenceId on Highlights results, which come back in data.highlights.

Local pages and files

Fetch a page on your machine before you deploy it. Enable local mode in the CLI or Node.js TypeScript SDK to get the same Markdown text or JSON object as a public URL Fetch.

Local mode requires authentication. Sign in with expandai login for the CLI, or set EXPAND_API_KEY for the SDK.

import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient()

const markdown = await expand.fetch({ url: 'http://localhost:3000' }, { local: true })
const page = await expand.fetchJson({ url: './dist' }, { local: true })
expandai fetch --local http://localhost:3000
expandai fetch

Choose what to share:

  • A localhost URL shares your running HTTP server, including other routes on the same port. 127.0.0.1, [::1], 0.0.0.0, and bare localhost:3000 are also accepted.
  • A file path shares only that file. Use a folder path if the page needs neighboring stylesheets, scripts, or images.
  • A folder path serves index.html and its assets. Hidden paths such as .env and .git, and symbolic links outside the folder, are excluded. Your home folder and folders containing it are refused. Choose a folder such as ./dist.

Relative paths start from the process's working directory. Asset URLs must be reachable from the public page. Local mode does not rewrite HTML or JavaScript that points to localhost or another local port.

The CLI or SDK gives Fetch a temporary public URL under expandusercontent.com automatically. Keep the process running until the call finishes. The connection closes on success, failure, or cancellation.

Warning
Anyone with the temporary URL can reach the selected content while the connection is open. Saved Snapshots remain shareable after it closes, so choose content you are willing to share.

Local mode supports fetch() and fetchJson() with a url in the Node.js TypeScript SDK, including ExpandService, and expandai fetch --local. Local HTTPS, Crawl, Batched Fetch, Python, and direct MCP local access are not supported.

Warning
In a server application, local mode reads files and services on that server. Never enable it for URLs or paths supplied by untrusted users.

Local input errors

Invalid inputs fail before Fetch starts, with a message explaining what to change. The CLI asks for --local when you pass a localhost HTTP URL without it. For a page that cannot load, check that your HTTP server is running at the address and port you supplied.

SDK callers can inspect ExpandClientLocalTargetError.reason and .input. Effect callers receive ExpandLocalTargetError. Reasons distinguish missing paths (NotFound), unreadable paths (PermissionDenied), public URLs (NotLocal), unsupported local protocols (UnsupportedProtocol), and unsafe folders (UnsafeDirectory). For failures after Fetch starts, see Errors & retries.

Include options

You choose what a Fetch returns in two places, in either mode:

  • The URL's ?include= parameter turns two layers on and off: State JSON and the Appendix. An empty ?include= turns State JSON off. ?include=appendix,statejson turns both on.
  • The request body's include.appendix and include.json do the same when the URL has no ?include=. Without either, you get State JSON but not the Appendix.
  • The rest of the body's include field asks for extra artifacts outside the document, such as a screenshot. Only JSON Mode can return one. In Markdown Mode, a requested screenshot is saved with the Snapshot but not returned.

Pick a format and what to include. The code below shows how to ask for it on each surface.

Always
Always
Always in Markdown text
JSON object only
import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient()

const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' })
expandai fetch https://news.ycombinator.com

# The Appendix, afterwards
expandai appendix <snapshotId>
// Arguments for the fetch tool
{
  "url": "https://news.ycombinator.com"
}
curl https://api.expand.ai/v1/fetch \
  -H "x-expand-api-key: $EXPAND_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com"
  }'

Response

---
url: https://news.ycombinator.com
snapshotId: ...
playground: https://expand.land/s/...
title: Hacker News
description: ...
appendix:
  url: https://expand.land/s/.../appendix
  links:
    sameDomain: 42
    external: 7
stateJson:
  moreTokens: 1200 # only when some State JSON was held back
---

# Hacker News

[appendix omitted: 14 links · ~320 tokens] {612}

1. Example story title
2. Another story title

## Expand State JSON

```json id=3
{ "stories": [ ... ] }
```

Not every field is shown. The full response is in the API reference.

Rules to remember:

  • Use statejson in URLs. Other spellings, such as stateJson and state-json, also work.
  • Unknown tokens are ignored.
  • ?include=appendix on its own turns State JSON off. List both to keep both.
  • When the URL has ?include=, even an empty one, it wins over the body's include.appendix and include.json.
  • A screenshot can cover the whole page. See Browser-rendered pages: Screenshots.
  • In a Batched Fetch, set options in the start request's body. Polling takes an include too, but only for what the run captured.

References to more

To keep responses small, Fetch sends a pointer instead of content that's large or secondary. Each pointer says what was held back and where to get it, so nothing is lost.

PointerWhat was held backHow to get it
appendix in the frontmatter, meta.appendix in the JSON objectThe AppendixOpen its url, or fetch with ?include=appendix
[appendix omitted: …] {612} in the Main MarkdownOne piece of the Appendix

Everything else the capture saved stays one request away: meta.resources links to it. See Progressive disclosure.

Next steps

  • Errors & retries: what each error means and whether to retry.
  • Progressive disclosure: read meta.resources and the artifacts behind a Snapshot.
  • Two ways to highlight: highlight a Fetch, choose what gets highlighted, and build citations.
  • Playground & replay: inspect Snapshots and citations visually.
  • API reference: exact schemas and nested include defaults.
PreviousFetch: Overview
NextBatched Fetch

On This Page

Two modesLocal pages and filesLocal input errorsInclude optionsReferences to moreNext steps
  …\n\n## Appendix\n\n- [new]…",
State JSON
"json": [
  { "stories": [ ... ] }
]
--local
./page.html
expandai fetch --local ./dist --format json
expandai appendix <snapshotId> --id 612
stateJson.moreTokensSome State JSON, about that many tokensRun Highlights on the Snapshot
A json entry with moreTokens and keys but no bodyThat entry, too large to sendRequest /s/<snapshotId>?id=<id>, or open it in Playground & replay
meta.markdown on a Highlights responseThe full Main MarkdownOpen its url