superagnt_
Map

Map a website's URLs

POST/map
priced per calljson responserest · mcp
01// about this endpoint

Map a website through agntdata to get a list of its URLs quickly. Optionally order results by relevance to a search term. Ideal for discovering a site's structure before scraping specific pages.

02// code samples

Call it over REST

Authenticate with a bearer token against https://api.agntdata.dev. Swap the placeholders for your key and parameter values.

curlbash
curl -s -X POST "https://api.agntdata.dev/map" \
  -H "Authorization: Bearer <YOUR_AGNTDATA_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{ "url": "<string>" }'
03// give it to your agent

Copy-paste prompt

Paste this into a coding agent. It connects the superagnt MCP server, finds the tool data_web_map, and runs a first call — with the REST fallback if MCP is unavailable.

agent prompttext
You are wiring up the agntdata "Web" API to use this endpoint: Map a website's URLs.

1. CONNECT over MCP (preferred). Add the superagnt MCP server, then let the client
   run OAuth on first tool call — no token to paste:
     claude mcp add --scope user --transport http superagnt https://mcp.superagnt.com/mcp
     claude mcp login superagnt
   MCP endpoint: https://mcp.superagnt.com/mcp
   After connecting, the tool you want is named exactly:
     data_web_map
   Call agnt_tools_search or agnt_tools_list_enabled to confirm it is available.

2. REST fallback (if you are not using MCP):
     POST https://api.agntdata.dev/map
     Header: Authorization: Bearer <YOUR_AGNTDATA_API_KEY>
   Required parameters:
    - url (string, required) — The base URL of the site to map.
   Optional parameters:
    - search (string, optional) — Order returned URLs by relevance to this query.
    - sitemap (string, optional) — How to use the site's sitemap. Default 'include'.
    - includeSubdomains (boolean, optional) — Include URLs on subdomains.
    - ignoreQueryParameters (boolean, optional) — Treat URLs that differ only by query string as the same URL.
    - limit (integer, optional) — Maximum number of URLs to return. Default 5000, max 100000.
    - timeout (integer, optional) — Request timeout in milliseconds.
    - location (object, optional) — Geographic proxy location and language settings.

3. COST & KEY. This endpoint is priced per call in credits, and each successful response reports its own cost in its meta.costCents field. Only successful calls are charged.
   Get an agntdata API key at https://app.agntdata.dev.

4. TEST. Connect the server (or set the key), discover the tool
   (data_web_map), run one minimal call with just the required parameters,
   and report back the shape of the JSON response (top-level fields).

Reference (machine-readable variant): https://superagnt.com/docs/apis/web/endpoints/map.md
mcp tool definitionjson
{
  "name": "map",
  "description": "Returns a list of URLs found on a website, quickly. Useful for discovering the structure of a site before scraping specific pages. Optionally order results by relevance to a `search` term.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "format": "uri",
        "description": "The base URL of the site to map."
      },
      "search": {
        "type": "string",
        "description": "Order returned URLs by relevance to this query."
      },
      "sitemap": {
        "type": "string",
        "enum": [
          "skip",
          "include",
          "only"
        ],
        "description": "How to use the site's sitemap. Default 'include'.",
        "default": "include"
      },
      "includeSubdomains": {
        "type": "boolean",
        "description": "Include URLs on subdomains.",
        "default": true
      },
      "ignoreQueryParameters": {
        "type": "boolean",
        "description": "Treat URLs that differ only by query string as the same URL.",
        "default": true
      },
      "limit": {
        "type": "integer",
        "description": "Maximum number of URLs to return. Default 5000, max 100000.",
        "minimum": 1,
        "maximum": 100000,
        "default": 5000
      },
      "timeout": {
        "type": "integer",
        "description": "Request timeout in milliseconds.",
        "minimum": 1000
      },
      "location": {
        "type": "object",
        "description": "Geographic proxy location and language settings.",
        "properties": {
          "country": {
            "type": "string",
            "description": "ISO 3166-1 alpha-2 country code."
          },
          "languages": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      }
    },
    "required": [
      "url"
    ]
  }
}
04// parameters
NameInTypeRequiredDescription
urlbodystringrequiredThe base URL of the site to map.
searchbodystringoptionalOrder returned URLs by relevance to this query.
sitemapbodystringoptionalHow to use the site's sitemap. Default 'include'.
includeSubdomainsbodybooleanoptionalInclude URLs on subdomains.
ignoreQueryParametersbodybooleanoptionalTreat URLs that differ only by query string as the same URL.
limitbodyintegeroptionalMaximum number of URLs to return. Default 5000, max 100000.
timeoutbodyintegeroptionalRequest timeout in milliseconds.
locationbodyobjectoptionalGeographic proxy location and language settings.
05// responses

The list of URLs discovered on the site.

200json
{
  "type": "object"
}
06// pricing
priced per call

This endpoint is priced per call and deducted from your agntdata balance; only a successful call is billed. Every billable response reports its own cost in meta.costCents.

Get a key at app.agntdata.dev.

07// related endpoints
08// more from Web

start calling

Point your client at https://mcp.superagnt.com/mcp and your agent has this endpoint, plus every other source on one balance.

Start free
site map URL discovery APIwebsite URL list APIcrawl planning APIAI agents web data APIweb scraping APIweb search API for AI agentsscrape to markdown API