API & MCP
Everything a person can do on Structured Page Extractor, a program or AI agent can do with the same inputs and get the identical result.
HTTP API
POST https://obscuriqpageextractor.usefulweb.app/api/v1/calculate– run Structured Page ExtractorGET https://obscuriqpageextractor.usefulweb.app/api/v1/tool– inputs, output schema, examples- OpenAPI 3.1: /openapi.json · /openapi.yaml
- Errors:
application/problem+json;422includes per-fielderrors.
Structured Page Extractor
Reads one public HTML page (or HTML you paste) with a small recipe of CSS selectors, text/attribute/regex rules and an optional same-site next-page link, and returns a table as JSON and CSV. Compares with an earlier run to show new, removed and changed rows, and suggests replacement selectors when a site changed. robots.txt is checked first; no JavaScript, cookies or logins.
| Field | Type | Default | Description |
|---|---|---|---|
url | text | — | Public page address. http(s) only. robots.txt is checked first; pages behind logins, paywalls or bot checks are not read. |
html | textarea | — | Or paste the page HTML. Used instead of the address; nothing is fetched. |
baseUrl | text | — | Address of the pasted page (for relative links) |
recipe | textarea | "{\n \"mode\": \"list\",\n \"item\": \"li.notice\",\n \"fields\": [\n {\n | Recipe (JSON). JSON: {"mode":"list","item":"li.notice","fields":[{"name":"title","selector":"a"},{"name":"link","selector":"a::attr(href)","type":"url"},{"name":"price","selector":".price","type":"number"}],"next":"a[rel=next]","maxPages":2,"key":"link"}. Field options: take text|own-text|attr (+ attr), regex (first group kept), type text|number|url, all (join every match). |
previous | textarea | — | Earlier run to compare (JSON export or CSV, optional) |
returnHtml | boolean | false | Include the first page’s HTML in the result (up to 1 MB, for a live preview) |
Example request
Notices list from pasted HTML
curl -X POST https://obscuriqpageextractor.usefulweb.app/api/v1/calculate -H 'content-type: application/json' -d '{"html":"<!doctype html><html lang=\"en\"><head><title>Northwind Notices — public register</title></head><body>\n<main><h1>Public notices</h1>\n<ul class=\"notices\">\n<li class=\"notice\"><a class=\"title\" href=\"/notices/101\">Road works on Harbour Street</a><span class=\"date\">2026-11-02</span><span class=\"fee\">€ 1.250,00</span></li>\n<li class=\"notice\"><a class=\"title\" href=\"/notices/102\">New opening hours: library</a><span class=\"date\">2026-11-05</span><span class=\"fee\">free</span></li>\n<li class=\"notice featured\"><a class=\"title\" href=\"/notices/103\">Tender: school roof repair</a><span class=\"date\">2026-11-09</span><span class=\"fee\">€ 84.000</span></li>\n</ul>\n<nav class=\"pager\"><a rel=\"next\" href=\"/notices?page=2\">Next</a></nav></main></body></html>","baseUrl":"https://notices.example/notices","recipe":"{\"mode\":\"list\",\"item\":\"li.notice\",\"fields\":[{\"name\":\"title\",\"selector\":\"a.title\",\"take\":\"text\",\"type\":\"text\"},{\"name\":\"link\",\"selector\":\"a.title\",\"take\":\"attr\",\"attr\":\"href\",\"type\":\"url\"},{\"name\":\"date\",\"selector\":\".date\",\"take\":\"text\",\"type\":\"text\",\"regex\":\"(\\\\d{4}-\\\\d{2}-\\\\d{2})\"},{\"name\":\"fee (EUR)\",\"selector\":\".fee\",\"take\":\"text\",\"type\":\"number\"}],\"next\":\"a[rel=next]\",\"maxPages\":1,\"key\":\"link\"}"}'
Every result contains
inputs (normalised, defaults applied), summary, result (validated against the published schema), units, assumptions, sources, warnings, computedAt and links.
MCP (Model Context Protocol)
- Streamable HTTP endpoint:
https://obscuriqpageextractor.usefulweb.app/mcp– protocol revisions 2026-07-28 (stateless) and 2025-11-25 / 2025-06-18 / 2025-03-26 (initialize handshake). - Tools:
obscuriq_page_extractor - Local stdio connector:
node src/mcp-stdio.ts --tool=obscuriq-page-extractor
{
"mcpServers": {
"obscuriq-page-extractor": { "type": "http", "url": "https://obscuriqpageextractor.usefulweb.app/mcp" }
}
}