Web scraping API for AI agents
Turn any public page into clean markdown or schema-shaped JSON, search the web and get the pages' content in the same call, and map a site to find the pages worth reading, from Claude, ChatGPT or Codex. In our test 40 scrapes of 10 real pages all succeeded, from $0.00144 a page.
Read-only. These tools fetch public web pages and search results; they never fill a form, sign in or post anything, and pages behind a login stay out of reach.
updated Nov 3, 2026
Copy the setup prompt.
Your agent does the rest.
Connect the superagnt MCP server at https://mcp.superagnt.com/mcp/web-scraping to this agent for me (I'll sign in through my browser). The setup guide for each kind of agent is at https://www.superagnt.com/install?facet=web-scraping» a new workspace starts with the Web scraping tools already on. On an existing workspace, your agent switches them on during setup.
What does each scraping option cost per page, and does it work?
We scraped the same 10 public pages four ways: docs, news, e-commerce and pricing pages, GitHub and Wikipedia. All 40 scrapes returned content and were billed; nothing failed.
| Option | Pages returned | Cost a page | Median time |
|---|---|---|---|
| Markdown (cached if fetched in the last 2 days) | 10 of 10 | $0.00144 | 1.0 seconds |
| Markdown, fresh (maxAge 0) | 10 of 10 | $0.00144 | 2.4 seconds |
| Markdown, enhanced anti-bot proxy | 10 of 10 | $0.00144 on these pages; up to $0.0072 when the anti-bot work is heavier | 3.2 seconds |
| JSON from a schema | 10 of 10, with the schema's fields | $0.0072 | 3.0 seconds |
| Search, 3 results | Titles, links and snippets | $0.00288 a search | 2.6 seconds |
| Search, 3 results with page content | Each result's markdown too | $0.0072 a search | 3.9 seconds |
sample · 10 public pages, 40 scrapes and 2 searches, Nov 3, 2026
What can your agent read from the web?
Any URL to clean markdown with navigation, cookie banners and boilerplate stripped, which is what a model can read.
Pass a JSON schema or a prompt and get typed fields back instead of prose, ready for a table.
One call runs a web search and can return each result's page content too, so the agent skips a round trip per link.
List a site's URLs fast, then scrape only the pages that matter.
JavaScript-heavy pages are rendered before extraction, with waits and an enhanced anti-bot mode per request.
A page scraped in the last two days comes back from cache by default; set maxAge to 0 for a fresh fetch.
data_web_scrapecalled by its own namedata_web_searchcalled by its own namedata_web_mapcalled by its own name
What does a real call return?
A run we recorded on superagnt, with the result trimmed to fit. Nothing on it is invented: the date, the wall time and the cost are the run's own.
asked Read superagnt's pricing page and tell me what the Starter plan includes.
- → call
data_web_scrape{ "url": "https://www.superagnt.com/pricing", "formats": ["markdown"], "onlyMainContent": true }← resulttrimmed"markdown": "## Every capability on every plan. You pick how much. ... Starter Every capability, sized for your first agents. $49/mo $20 data credits included / mo - 1 workspace · 3 deployed agents - 10 schedules · 3 queues - Database: 10 tables, 25,000 rows - 60 requests / min ...", "metadata": { "title": "superagnt Pricing: Plans From $49 a Month", "statusCode": 200 }
the same call over httpcurl · python
curl -X POST "https://api.superagnt.com/v1/data/web/scrape" \
-H "Authorization: Bearer $SUPERAGNT_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.superagnt.com/pricing","onlyMainContent":true}'import os, requests
r = requests.post(
"https://api.superagnt.com/v1/data/web/scrape",
json={"url": "https://www.superagnt.com/pricing", "onlyMainContent": True},
headers={"Authorization": f"Bearer {os.environ['SUPERAGNT_API_KEY']}"},
)
print(r.json())Why markdown instead of raw HTML?
Because the model pays for every character it reads. On the same 10 pages the markdown was 25 times smaller than the raw HTML at the median: 17,000 characters against 348,000. The gap ran from 2.4 times on a plain docs page to 107 times on an e-commerce home page full of scripts and styles.
sample · 10 pages fetched as markdown and raw HTML in one call each, Nov 3, 2026
Web search API or web scraping API: which does an agent need?
Both, for different questions. Search finds pages you don't know yet; scraping reads a page you already have the address of; mapping lists a site's pages before you choose which to read. With page content turned on, one search call returned 3 results and their markdown for $0.0072, instead of a search and three separate scrapes. Use a real browser only when the job is to act on a page, such as filling a form: this source reads.
Is the content live?
Fresh on request. By default a page scraped in the last two days comes back from cache, which is faster: about 1 second against 2.4 for a fresh fetch in our test. Set maxAge to 0 when the page changes often, such as a price or a stock level.
What do people use it for?
Research that cites real pages
The agent searches, reads the page content that comes back with the results, and answers with links and quotes it retrieved rather than what it remembers.
Pricing and positioning watch
A schedule scrapes competitor pricing pages into a schema, writes each run to the workspace database, and reports what changed since last week.
Enrichment from a company site
Given a domain, the agent maps the site, scrapes the about and careers pages, and pairs what it finds with company enrichment and SEO data.
What does it cost?
Heavier calls cost more; each response reports its own cost.
About 2,083 calls at the lowest price.
Only a call that returns data is billed: a miss, a refusal or an upstream failure costs nothing. One credit balance covers this source and every other one, so there is no per-source subscription and no upstream account of yours to keep alive. After the trial, plans are Starter $49 a month ($20 of data credit included), Growth $149 a month ($90) and Scale $399 a month ($300). Requests per minute: Starter 60, Growth 300 and Scale 600.
What do people ask before they wire it up?
01Which web scraping API is best for AI agents?
One that returns what a model can read (markdown or schema-shaped JSON), handles rendered and protected pages, and bills per page you can predict. In our test of 10 varied pages every option succeeded, at $0.00144 a page for markdown and $0.0072 for JSON.
02What is the difference between a web search API and a web scraping API?
A search API finds pages that match a query and returns titles, links and snippets; a scraping API reads one page you already know. Search with page content does both in one call.
03Does a scraping tool need MCP support?
For an agent in Claude, ChatGPT, Codex or Cursor, yes: MCP is how it calls the tool without code of yours in between. These tools work over MCP and over plain HTTP with the same parameters.
04When should an agent use a real browser instead of an API?
When it has to act on the page: sign in, click through a flow or fill a form. To read a public page, an API is faster and cheaper, and it renders JavaScript pages before extracting them.
05Do I still need proxies or my own scraping account?
No. Rendering and anti-bot handling are part of the call, with an enhanced mode per request, and one superagnt key covers this and every other source.
06How do I know the scraped data is usable, and how should I compare prices?
Compare cost per page you can use, not per request. Ask for a JSON schema and check the fields come back filled: they did on 10 of 10 pages in our test. And count characters, since markdown was 25 times smaller than raw HTML at the median.