Developers / Outway API
Collect public pages.
Read every outcome.
Queue scrape, batch, crawl and URL-map jobs against the production API. Inspect job state, partial results and saved capture artifacts.
Sign in to your workspace, choose a project and create a scoped API key. Keep the token in your local environment or secret manager. The API also accepts a verified OpenID Connect (OIDC) access token. Your account, project permissions and collection allowance determine which requests you can run. Review authentication and access →
Queue your first scrape.
Send a public HTTP or HTTPS URL to https://api.outway.dev. New jobs return immediately with an ID; poll that job to read its state and capture ID. Set an Idempotency-Key when you may retry a request. Reuse the same key only with the same operation and payload.
curl --request POST "https://api.outway.dev/v1/projects/PROJECT_ID/scrapes" \
--header "Authorization: Bearer $OUTWAY_TOKEN" \
--header "Content-Type: application/json" \
--header "Idempotency-Key: 8d5014ed-2a13-4dab-914f-571053165a23" \
--data '{
"url": "https://example.com"
}'Optional rendering and request limits
- Browser rendering is optional; wait time is capped at 5 seconds.
- Each scrape can request up to 10 bounded click or scroll actions.
cacheMode: revalidateasks upstream HTTP caches to revalidate. It does not guarantee that the origin is fresh.- A conflicting reuse of an idempotency key returns HTTP 409.
Keep the Bearer token out of browser code, public repositories and logs.
TypeScript Quickstart
Retrieve your first Markdown capture.
Download the example below and run it with Node.js 24. Set OUTWAY_TOKEN and OUTWAY_PROJECT_ID from your provisioned preview workspace, then run node first-capture.ts. The script submits a public page, polls with backoff for up to ten minutes, verifies the artifact digest, and prints Markdown. It needs no npm packages. Keep credentials in your server environment, never in a browser bundle or committed file.
Download first-capture.ts ↓ Download OpenAPI ↓ Contract source revision
The complete example below handles polling, retries and integrity checks. This excerpt shows how to run it.
if (import.meta.main) {
firstCapture({
token: process.env.OUTWAY_TOKEN ?? "",
projectId: process.env.OUTWAY_PROJECT_ID ?? "",
url: process.env.OUTWAY_URL ?? "https://example.com",
// Preserve this value when retrying the same submission and payload.
idempotencyKey: process.env.OUTWAY_IDEMPOTENCY_KEY || randomUUID(),
}).then(({ markdown }) => process.stdout.write(markdown))
.catch((error) => { console.error(error.message); process.exitCode = 1; });
}View the complete Node.js example
import { randomUUID, createHash } from "node:crypto";
import { setTimeout as sleep } from "node:timers/promises";
type Job = { id: string; state: string; captureId: string | null };
type Artifact = { id: string; kind: string; sha256: string };
// Node.js 24: run `node first-capture.ts` with credentials in the environment.
export async function firstCapture({
token,
projectId,
url = "https://example.com",
idempotencyKey = randomUUID(),
fetcher = fetch,
timeoutMs = 600_000,
pollMs = 1_000,
}: {
token: string;
projectId: string;
url?: string;
idempotencyKey?: string;
fetcher?: typeof fetch;
timeoutMs?: number;
pollMs?: number;
}) {
if (!token || !projectId) throw new Error("Set OUTWAY_TOKEN and OUTWAY_PROJECT_ID.");
const signal = AbortSignal.timeout(timeoutMs);
const base = "https://api.outway.dev";
let jobId: string | undefined;
const request = async (path: string, init: RequestInit = {}) => {
const response = await fetcher(base + path, {
...init,
headers: { Authorization: `Bearer ${token}`, ...init.headers },
redirect: "error",
signal,
});
if (!response.ok) {
// Avoid echoing response bodies or credentials into logs.
throw new Error(`Outway request failed (HTTP ${response.status}).`);
}
return response;
};
try {
let job: Job = await (await request(`/v1/projects/${encodeURIComponent(projectId)}/scrapes`, {
method: "POST",
headers: { "Content-Type": "application/json", "Idempotency-Key": idempotencyKey },
body: JSON.stringify({ url }),
})).json();
jobId = job.id;
let delay = pollMs;
while (["queued", "running", "waiting_retry"].includes(job.state)) {
await sleep(delay, undefined, { signal });
job = await (await request(`/v1/jobs/${encodeURIComponent(jobId)}`)).json();
delay = Math.min(delay * 2, 5_000);
}
if (job.state !== "succeeded" || !job.captureId) {
throw new Error(`Job ${jobId} ended with state ${job.state}; inspect its status before retrying.`);
}
const capturePath = `/v1/captures/${encodeURIComponent(job.captureId)}`;
const artifacts: Artifact[] = await (await request(`${capturePath}/artifacts`)).json();
const artifact = artifacts.find((item) => item.kind === "normalized_markdown");
if (!artifact) throw new Error(`Capture ${job.captureId} has no Markdown artifact.`);
const response = await request(`${capturePath}/artifacts/${encodeURIComponent(artifact.id)}/content`);
const bytes = Buffer.from(await response.arrayBuffer());
if (createHash("sha256").update(bytes).digest("hex") !== artifact.sha256) {
throw new Error("Artifact integrity check failed.");
}
return { jobId, captureId: job.captureId, markdown: bytes.toString("utf8") };
} catch (error) {
if (signal.aborted) {
throw new Error(`Stopped waiting${jobId ? ` for job ${jobId}` : " for submission"}. The server job may continue; inspect its status or cancel it explicitly.`);
}
throw error;
}
}
if (import.meta.main) {
firstCapture({
token: process.env.OUTWAY_TOKEN ?? "",
projectId: process.env.OUTWAY_PROJECT_ID ?? "",
url: process.env.OUTWAY_URL ?? "https://example.com",
// Preserve this value when retrying the same submission and payload.
idempotencyKey: process.env.OUTWAY_IDEMPOTENCY_KEY || randomUUID(),
}).then(({ markdown }) => process.stdout.write(markdown))
.catch((error) => { console.error(error.message); process.exitCode = 1; });
}Retries and failures
Set OUTWAY_IDEMPOTENCY_KEY to a stable value when retrying the same submission and URL; use a new value for a new capture. The example does not automatically resubmit an uncertain request. A local timeout stops waiting, not the server job. Use the reported job ID to inspect its status or call the cancellation endpoint.
- 401/403: check token validity, account verification and project permissions.
- 404: check that the project, job or artifact belongs to your workspace.
- 429: your organization has reached its active-job limit. Wait for existing work to finish before retrying; this limit is separate from billing quotas.
- 409: an idempotency key was reused with a different request.
- Failed or missing Markdown: inspect the job and capture details; never treat an absent artifact as a successful empty result.
Current API surface
Collection and result endpoints.
| Method and path | Purpose and bound |
|---|---|
POST /v1/projects/{projectId}/scrapes | Queue one public-page scrape. |
POST /v1/projects/{projectId}/batches | Queue up to 100 unique URLs; each URL gets an independent job. |
GET /v1/batches/{batchId} | Read batch progress and ordered per-URL outcomes. |
POST /v1/projects/{projectId}/crawls | Queue a same-origin crawl; depth 0–3 and 1–500 pages. |
POST /v1/projects/{projectId}/maps | Discover same-origin URLs with the same depth/page bounds. Map results do not create captures. |
GET /v1/jobs/{jobId} | Poll job state, retry time, capture ID and last error. |
GET /v1/projects/{projectId}/jobs?limit=20&before={jobId} | List newest-first project jobs; 20 by default, up to 100 per page. |
POST /v1/jobs/{jobId}/cancel | Cancel an active job. Repeating the request is safe. |
GET /v1/projects/{projectId}/captures | List capture history with the same newest-first cursor pattern. |
POST /v1/captures/{captureId}/extract | Apply up to 20 typed CSS fields to saved HTML, with source evidence and explicit missing values. |
GET /v1/captures/{captureId}/audit | Compute 18 saved-metadata checks; no new crawl or persisted report. |
GET /v1/captures/{captureId} | Read capture metadata. |
GET /v1/captures/{captureId}/artifacts | List available artifacts. |
GET /v1/captures/{captureId}/artifacts/{artifactId}/content | Download integrity-checked artifact content. |
Collection requests are asynchronous. Saved extraction and audit reads return directly. Poll the returned job or crawl/batch ID; crawl and map responses preserve per-page success, retry, failure and skipped states.
Bounded crawl
Set depth and page limits.
Crawl and map stay on the starting URL’s origin, follow robots policy and return partial page outcomes. The limits below apply to each request; they are not customer-specific usage quotas.
curl --request POST "https://api.outway.dev/v1/projects/PROJECT_ID/crawls" \
--header "Authorization: Bearer $OUTWAY_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"url": "https://example.com/docs",
"maxDepth": 2,
"maxPages": 50
}'Batch collection
Queue independent URLs.
Batch requests accept up to 100 unique URLs after normalization. Each URL receives its own job and result. Use the batch status endpoint to read per-URL outcomes.
curl --request POST "https://api.outway.dev/v1/projects/PROJECT_ID/batches" \
--header "Authorization: Bearer $OUTWAY_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"urls": [
"https://example.com/one",
"https://example.com/two"
]
}'Search Data pilot
Review imported SERPs and rank history.
The first pilot is Google organic results for the United States, in English on desktop. Import existing observations with POST /v1/projects/{projectId}/serp-snapshots, using format (outway-serp-v1 or dataforseo-google-organic-v3) and a JSON payload. Native observations include engine, country, language, device, keyword, observedAt, depth and organic results with rank, absoluteRank, URL, title and description.
List summaries, reopen or delete a snapshot. Up to 200 snapshots per project, 100 organic results each and 2 MiB per import; repeated normalized imports reuse the same record. Read GET /v1/projects/{projectId}/rank-history?keyword=...&domain=... for exact-host positions over time; includeSubdomains=true includes real subdomains. Positive improvement means a better organic position. Missing or incompatible observations cannot prove a ranking change.
All evidence is labelled imported and unverified. The optional DataForSEO response adapter makes no provider calls and charges nothing. Live collection, scheduled tracking, keyword volumes and backlink data are not supplied by this feature.
Automatic bounded crawl audits
Read worker reports and compare evidence.
The public schema includes the current collection, workspace, subscription and evidence contracts, synchronized with the services source recorded in contract provenance.
GET /v1/projects/{projectId}/site-audits returns the latest 50 worker report summaries and hasMore. Read a report with GET /v1/crawls/{crawlId}/audit and compare an earlier same-project crawl with GET /v1/crawls/{crawlId}/audit-comparison?baselineCrawlId=....
These automatic 18-rule reports differ from older user-saved audit reports. Coverage shows recorded, audited, complete and unavailable pages; whole-site coverage is never verified. Root, scope, budgets, rules and observation times must be compatible. Missing, failed or partial evidence never establishes resolution. Older, cancelled or unavailable reports return 409, not an empty audit.
SDK methods: listSiteAudits, getSiteAudit and compareSiteAudits. Migration a13de672fc8b is required before new API/worker rollout. Source publication alone does not verify migration completion or live availability.
Imported domain evidence
GET /v1/projects/{projectId}/domain-intelligence?domain=example.com&competitor=other.example&includeSubdomains=true compares two hostnames over at most 200 retained imported snapshots. SDK method: getDomainIntelligence. Keep the exact keyword/provider/depth universe visible. Shared observations, observed gaps, exclusive results and landing-page evidence carry source snapshot IDs and timestamps. Equal-time observations preserve ambiguous null values. Imported and unverified data does not establish global keyword coverage; absent positions are unknown. Volume, CPC, traffic and difficulty are unavailable.
Saved Web Data
Extract fields, read documents, inspect evidence.
Send a field recipe to POST /v1/captures/{captureId}/extract after capturing an HTML page. The API reads saved HTML without new website or model calls. Use valueType for strings, finite numbers, true/false booleans or HTTP(S) URLs; optional attribute reads markup attributes and all collects an array.
{
"fields": [
{
"name": "title",
"selector": "h1",
"required": true
},
{
"name": "links",
"selector": "a",
"attribute": "href",
"valueType": "url",
"all": true
}
]
}Inspect valid, completeCapture, field statuses and source provenance separately. Required missing values, invalid conversions and truncation need review. Recipes allow up to 20 fields, 100 matches per field and 2,000 characters per value. CSS pseudo-classes/escapes, scripts and model inference are unsupported.
Reusable project recipes
List or save recipes with GET/POST /v1/projects/{projectId}/extraction-recipes, and delete an entry with DELETE /v1/projects/{projectId}/extraction-recipes/{recipeId}. Save title and recipe; up to 50 immutable entries, 16 KB each. Identical recipes reuse their existing ID and title. The dashboard supports save, load, portable import and confirmed deletion.
PDF text
Submit a public PDF through the existing scrape endpoint. Text PDFs produce page-numbered document_metadata and normalized Markdown when extraction succeeds. Limits are 5 MiB input, 50 pages and 1 MiB extracted text. Image-only files, encrypted/malformed input and limits have explicit states; OCR supports English, German, French, Spanish, Italian, Portuguese and Dutch (up to three selected languages), and attempts up to 10 textless pages, with a 60-second OCR deadline and bounded grayscale rasterization. Native text is preserved. Per-page methods and OCR failures are recorded; recognition can be inaccurate. Layout reconstruction is not performed. Download the original file or available page evidence through artifact routes.
Server audits and exact monitor baselines
The saved capture audit API computes 18 observations, including markup, robots, image-alt, social and JSON-LD checks, with evidence and uncertain coverage. The on-demand endpoint does not save a report. New HTML monitor runs automatically retain the newest 100 audit reports and exact issue comparisons per monitor. Only complete, compatible evidence can establish resolution; old, failed or PDF runs can have no report. Audits do not produce ranking, accessibility or performance scores. Monitor history includes the exact recorded baseline capture/hash/time for new successful runs; older runs remain explicitly unrecorded. Retention can make historical artifacts unavailable.
The TypeScript client and local MCP integration expose these saved-data tools. Returned page content is untrusted data, never instructions. Authentication and preview access requirements remain unchanged.
Experimental native Search
Queued collection with explicit evidence.
POST /v1/projects/{projectId}/search-tasks accepts a keyword and optional transport (http or browser). It returns a normal Outway job. Poll the job, list its capture artifacts, then download search_results for structured evidence and the raw artifact for inspection.
This first-party path uses a shared source budget, regional worker eligibility and circuit breakers for blocked sources or unfamiliar layouts. Collection is disabled until an operator configures and validates the source network. A US request parameter or worker-region label is not proof of US localization. Recognized result order is experimental; imported rank history remains separate. This is not a Google API integration, DataForSEO reseller, or promise of verified live rankings.
Workspace settings
Inspect API capabilities and rename projects.
GET /v1/capabilities returns authenticated API admission settings, effective project/collection permissions and operational limits. Responses are private and not cached. Enabled API settings do not establish worker readiness, live search access or a paid entitlement.
PATCH /v1/projects/{projectId} accepts {"name":"Research","expectedName":"Original"} with projects:write permission. The current name must match exactly; stale edits return 409 and should reload before saving. Names use 1–160 Unicode characters without controls after trimming. Project IDs, retention, schedules and saved evidence are preserved. The dashboard exposes these controls in Settings.
Recurring collection
Schedule bounded crawls and search jobs.
/v1/projects/{projectId}/collection-schedules supports paused or enabled schedules, interval updates and latest-run history. Crawls stay on the exact origin with the usual page/depth limits. Intervals range from one hour to thirty days; active runs prevent overlap and missed intervals coalesce. Search activation retains the native collection gate. Pause or delete stops future scheduling; cancel an existing job separately. The dashboard can reopen scheduled crawls for on-demand audits.
Subscription and usage preview
GET /v1/billing/preview returns workspace diagnostic usage and a draft monthly subscription catalog. Successful page captures and mapped URLs record one diagnostic unit; successful native search captures record two diagnostic units. Failed/incomplete captures use zero. Metering begins when installed and survives evidence expiry. Payments, included-allowance enforcement and overages are inactive; this preview is not an invoice or active subscription.
Result handling
Keep partial results visible.
- Job states include queued, running, waiting_retry, succeeded, partial, failed and cancelled.
- A crawl or map is partial when some pages succeeded and others failed; failed page details remain in the response.
- Job and capture history use newest-first pagination. Pass the last item ID as the
beforecursor. - Capture artifacts include a SHA-256 digest; content downloads are integrity-checked.
Not in this API contract
Arbitrary schema inference is not part of native deterministic extraction. Optional model-assisted analysis has separate configuration, resource-use consent and evidence limitations. Native Search collection is experimental and disabled by default. Deterministic CSS extraction, native PDF text, saved capture audits and monitor/account routes are documented; production availability still requires preview validation. Imported SERP observations and rank history support the selected Google US/en/desktop pilot; scheduled collection is documented, while verified rank tracking, paid activation and provider acceptance remain separate.
The API publishes timestamps and configurable collection allowances; these do not establish a freshness SLA or measured capacity. See the versioned OpenAPI contract ↗.