Outway documentation

Crawl and map

Explore websites with bounded crawls and discover URLs with Outway map runs.

Crawl a website

Submit POST /v1/projects/{projectId}/crawls:

{"url":"https://example.com","maxDepth":1,"maxPages":20}

Set explicit bounds. Crawls support a maximum depth of three and a maximum of 500 pages. The requested page budget counts toward allowance; it is not simply the number of successful captures.

Render JavaScript pages

Add "render": {"waitAfterLoadMs": 1000} to a crawl request to use the native browser. Rendered crawls allow up to 50 pages and depth three. Each page saves rendered HTML, a screenshot, Markdown and collection metadata.

Scope and robots checks apply before every browser request, including redirects and assets. Out-of-scope assets can make a page incomplete; inspect partial results before using them in an audit. Existing per-page time, request and byte limits still apply. Map runs do not support rendering.

Inspect progress

Use the returned run identifier with GET /v1/crawls/{crawlId}. Inspect individual page outcomes and captures. A partial run can contain useful evidence and failed pages; do not describe it as complete coverage.

Discover URLs

Map runs discover site URLs through POST /v1/projects/{projectId}/maps. Inspect the required request fields in the API reference and read the result with GET /v1/maps/{mapId}. URL discovery does not guarantee a full capture of every URL.

Plan collection

Start with a small bounded run, inspect coverage and errors, then increase the budget deliberately. Neither crawl nor map guarantees access to authenticated, blocked or undiscoverable pages.