Smart Fetch (regional fetch)

POST https://tools.yukai.uk/api/v1/smart-fetch

Fetch a URL as clean Markdown from a chosen region. One URL in, clean Markdown (and/or HTML) plus metadata out, fetched from the region you pick (us, sg, eu, tw) or, with region=auto, from the cheapest network that gets through: Cloudflare edge, Phoenix, then a real Chrome browser for JavaScript pages. Honest by design: one User-Agent that names TidyTools on every request, robots.txt honored by default, no logins, no cookies, no CAPTCHA solving, no proxies pretending to be visitors; when a site refuses one network only the region changes, never the identity. Error pages served with HTTP 200 (soft 404, access denied, "not available in your region", cookie walls) are reported as failures, not returned as content. attempts shows every region and mode tried. Typical time 0.5-3 s; up to 55 s when every fallback is used.

Cost

Credits: 9 per HTTP page, +18 through a regional node (us, sg, eu, tw), 27 per browser-rendered page. 27 credits are reserved during the call and the difference is returned. Failed calls are free. Source price: smart-fetch/page-http.

Input

Send a JSON body (POST) or the same fields as a query string (GET).

Required: url.

Field	Type	Default	Description
`url` required	string		Page URL. A bare domain such as `example.com` is accepted (https is assumed). Private, local and cloud-metadata addresses are refused.
`region`	string	`"auto"`	Where the request leaves from. `auto` (default): Cloudflare edge, then Phoenix (US) when a site refuses it, then a real browser for JavaScript pages or when everything was refused. `us` Phoenix, `sg` Singapore, `eu` London, `tw` Taiwan (residential line), `cf` Cloudflare edge. A named region is tried alone (no silent substitute) unless `fallback=true`.
`mode`	string	`"auto"`	`auto` (default): HTTP first, a real Chrome browser when the page is a JavaScript shell or refused over HTTP (region `auto` or `cf` only). `http`: HTTP only. `browser`: browser only (regions `auto`, `cf` and `tw`; `tw` renders in a Taiwan Chrome).
`format`	string	`"markdown"`	`markdown` (default), `html`, or `both`. `metadata` is always returned.
`fallback`	boolean		Try other regions when the first is refused or fails. Default `true` for region `auto`, `false` for a named region (so the answer really comes from that region).
`respectRobots`	boolean	`true`	Honor the site's robots.txt for our crawler token. `false` fetches pages robots.txt disallows.
`maxChars`	integer	`200000`	Cut Markdown and HTML at this many characters (`truncated` tells you).
`timeoutMs`	integer		Timeout per attempt in milliseconds. Unset: 12 s per HTTP attempt with fallback (30 s for a single named region). The whole call is limited to 55 s.
`waitUntil`	string	`"load"`	Browser pages: when the page counts as loaded.
`includeLinks`	boolean	`false`	Also return the absolute links found on the page in `metadata.links`.
`mainContent`	boolean	`false`	Keep only the main article/content block (readability-style).
`removeBoilerplate`	boolean	`true`	Drop navigation, header, footer and similar page chrome.
`cssSelector`	string		Keep only the part of the page matching this CSS selector (e.g. `article`).
`removeSelectors`	string		Comma-separated CSS selectors removed before conversion.

Examples

Replace tt_live_XXXX with your key from the pricing page.

curl

curl -i -X POST "https://tools.yukai.uk/api/v1/smart-fetch" \
  -H "Authorization: Bearer tt_live_XXXX" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

JavaScript (fetch)

const res = await fetch("https://tools.yukai.uk/api/v1/smart-fetch", {
  method: "POST",
  headers: { "Authorization": "Bearer tt_live_XXXX", "Content-Type": "application/json" },
  body: JSON.stringify({"url":"https://example.com"}),
});
console.log(res.status, res.headers.get("x-credits-charged"), res.headers.get("x-credits-balance"));
console.log(await res.json());

Python (requests)

import requests

r = requests.post(
    "https://tools.yukai.uk/api/v1/smart-fetch",
    headers={"Authorization": "Bearer tt_live_XXXX"},
    json={"url": "https://example.com"},
    timeout=120,
)
print(r.status_code, r.headers.get("x-credits-charged"), r.headers.get("x-credits-balance"))
print(r.json())

Response

JSON with "success": true and the result fields; on failure {"success": false, "error": {"code", "message", "retryable"}} and no credits are used. Every answer carries x-credits-charged (credits used) and x-credits-balance (balance after the call); check the balance any time with GET https://tools.yukai.uk/v1/billing/balance.

Full field list and response schema: OpenAPI spec smart-fetch (its servers entry is the marketplace gateway; call https://tools.yukai.uk with your key).

← All routes · Pricing