superagnt

Web scraping API for AI agents

Turn any public page into clean markdown or schema-shaped JSON, search the web and get the pages' content in the same call, and map a site to find the pages worth reading, from Claude, ChatGPT or Codex. In our test 40 scrapes of 10 real pages all succeeded, from $0.00144 a page.

// data source

Read-only. These tools fetch public web pages and search results; they never fill a form, sign in or post anything, and pages behind a login stay out of reach.

updated Nov 3, 2026

Copy the setup prompt.
Your agent does the rest.

works with any agent+ 8 more
Connect the superagnt MCP server at https://mcp.superagnt.com/mcp/web-scraping to this agent for me (I'll sign in through my browser). The setup guide for each kind of agent is at https://www.superagnt.com/install?facet=web-scraping
or follow the detailed setup instructions

» a new workspace starts with the Web scraping tools already on. On an existing workspace, your agent switches them on during setup.

01// compared

What does each scraping option cost per page, and does it work?

We scraped the same 10 public pages four ways: docs, news, e-commerce and pricing pages, GitHub and Wikipedia. All 40 scrapes returned content and were billed; nothing failed.

OptionPages returnedCost a pageMedian time
Markdown (cached if fetched in the last 2 days)10 of 10$0.001441.0 seconds
Markdown, fresh (maxAge 0)10 of 10$0.001442.4 seconds
Markdown, enhanced anti-bot proxy10 of 10$0.00144 on these pages; up to $0.0072 when the anti-bot work is heavier3.2 seconds
JSON from a schema10 of 10, with the schema's fields$0.00723.0 seconds
Search, 3 resultsTitles, links and snippets$0.00288 a search2.6 seconds
Search, 3 results with page contentEach result's markdown too$0.0072 a search3.9 seconds

sample · 10 public pages, 40 scrapes and 2 searches, Nov 3, 2026

02// what is inside

What can your agent read from the web?

Scrape to markdown

Any URL to clean markdown with navigation, cookie banners and boilerplate stripped, which is what a model can read.

Structured extraction

Pass a JSON schema or a prompt and get typed fields back instead of prose, ready for a table.

Search with content

One call runs a web search and can return each result's page content too, so the agent skips a round trip per link.

Site mapping

List a site's URLs fast, then scrape only the pages that matter.

Rendered and protected pages

JavaScript-heavy pages are rendered before extraction, with waits and an enhanced anti-bot mode per request.

Fresh or cached

A page scraped in the last two days comes back from cache by default; set maxAge to 0 for a fresh fetch.

tool names3 toolsas a new server lists them
  • data_web_scrapecalled by its own name
  • data_web_searchcalled by its own name
  • data_web_mapcalled by its own name
03// recorded run

What does a real call return?

A run we recorded on superagnt, with the result trimmed to fit. Nothing on it is invented: the date, the wall time and the cost are the run's own.

recorded runrecorded Nov 3, 2026, 16:54 UTC · wall time 1.6 s · 1 call · 0.14¢

asked Read superagnt's pricing page and tell me what the Starter plan includes.

  1. → call data_web_scrape
    { "url": "https://www.superagnt.com/pricing", "formats": ["markdown"], "onlyMainContent": true }
    ← resulttrimmed
    "markdown": "## Every capability on every plan. You pick how much. ...
    Starter
    Every capability, sized for your first agents.
    $49/mo
    $20 data credits included / mo
    - 1 workspace · 3 deployed agents
    - 10 schedules · 3 queues
    - Database: 10 tables, 25,000 rows
    - 60 requests / min ...",
    "metadata": { "title": "superagnt Pricing: Plans From $49 a Month", "statusCode": 200 }
the same call over httpcurl · python
curl
curl -X POST "https://api.superagnt.com/v1/data/web/scrape" \
  -H "Authorization: Bearer $SUPERAGNT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.superagnt.com/pricing","onlyMainContent":true}'
python
import os, requests

r = requests.post(
    "https://api.superagnt.com/v1/data/web/scrape",
    json={"url": "https://www.superagnt.com/pricing", "onlyMainContent": True},
    headers={"Authorization": f"Bearer {os.environ['SUPERAGNT_API_KEY']}"},
)
print(r.json())
04// measured

Why markdown instead of raw HTML?

Because the model pays for every character it reads. On the same 10 pages the markdown was 25 times smaller than the raw HTML at the median: 17,000 characters against 348,000. The gap ran from 2.4 times on a plain docs page to 107 times on an e-commerce home page full of scripts and styles.

sample · 10 pages fetched as markdown and raw HTML in one call each, Nov 3, 2026

05// answered

Web search API or web scraping API: which does an agent need?

Both, for different questions. Search finds pages you don't know yet; scraping reads a page you already have the address of; mapping lists a site's pages before you choose which to read. With page content turned on, one search call returned 3 results and their markdown for $0.0072, instead of a search and three separate scrapes. Use a real browser only when the job is to act on a page, such as filling a form: this source reads.

06// answered

Is the content live?

Fresh on request. By default a page scraped in the last two days comes back from cache, which is faster: about 1 second against 2.4 for a fresh fetch in our test. Set maxAge to 0 when the page changes often, such as a price or a stock level.

07// what it gets used for

What do people use it for?

01

Research that cites real pages

The agent searches, reads the page content that comes back with the results, and answers with links and quotes it retrieved rather than what it remembers.

02

Pricing and positioning watch

A schedule scrapes competitor pricing pages into a schema, writes each run to the workspace database, and reports what changed since last week.

03

Enrichment from a company site

Given a domain, the agent maps the site, scrapes the about and careers pages, and pairs what it finds with company enrichment and SEO data.

08// cost

What does it cost?

from, per call
$0.001

Heavier calls cost more; each response reports its own cost.

from, per 1,000 calls
$1.44
free trial, no card
$3

About 2,083 calls at the lowest price.

Only a call that returns data is billed: a miss, a refusal or an upstream failure costs nothing. One credit balance covers this source and every other one, so there is no per-source subscription and no upstream account of yours to keep alive. After the trial, plans are Starter $49 a month ($20 of data credit included), Growth $149 a month ($90) and Scale $399 a month ($300). Requests per minute: Starter 60, Growth 300 and Scale 600.

09// questions

What do people ask before they wire it up?

01Which web scraping API is best for AI agents?

One that returns what a model can read (markdown or schema-shaped JSON), handles rendered and protected pages, and bills per page you can predict. In our test of 10 varied pages every option succeeded, at $0.00144 a page for markdown and $0.0072 for JSON.

02What is the difference between a web search API and a web scraping API?

A search API finds pages that match a query and returns titles, links and snippets; a scraping API reads one page you already know. Search with page content does both in one call.

03Does a scraping tool need MCP support?

For an agent in Claude, ChatGPT, Codex or Cursor, yes: MCP is how it calls the tool without code of yours in between. These tools work over MCP and over plain HTTP with the same parameters.

04When should an agent use a real browser instead of an API?

When it has to act on the page: sign in, click through a flow or fill a form. To read a public page, an API is faster and cheaper, and it renders JavaScript pages before extracting them.

05Do I still need proxies or my own scraping account?

No. Rendering and anti-bot handling are part of the call, with an enhanced mode per request, and one superagnt key covers this and every other source.

06How do I know the scraped data is usable, and how should I compare prices?

Compare cost per page you can use, not per request. Ask for a JSON schema and check the fields come back filled: they did on 10 of 10 pages in our test. And count characters, since markdown was 25 times smaller than raw HTML at the median.

10// keep going

What else can your agent reach?