Map a website's URLs
/mapMap a website through agntdata to get a list of its URLs quickly. Optionally order results by relevance to a search term. Ideal for discovering a site's structure before scraping specific pages.
Call it over REST
Authenticate with a bearer token against https://api.agntdata.dev. Swap the placeholders for your key and parameter values.
curl -s -X POST "https://api.agntdata.dev/map" \
-H "Authorization: Bearer <YOUR_AGNTDATA_API_KEY>" \
-H "Content-Type: application/json" \
-d '{ "url": "<string>" }'Copy-paste prompt
Paste this into a coding agent. It connects the superagnt MCP server, finds the tool data_web_map, and runs a first call — with the REST fallback if MCP is unavailable.
You are wiring up the agntdata "Web" API to use this endpoint: Map a website's URLs.
1. CONNECT over MCP (preferred). Add the superagnt MCP server, then let the client
run OAuth on first tool call — no token to paste:
claude mcp add --scope user --transport http superagnt https://mcp.superagnt.com/mcp
claude mcp login superagnt
MCP endpoint: https://mcp.superagnt.com/mcp
After connecting, the tool you want is named exactly:
data_web_map
Call agnt_tools_search or agnt_tools_list_enabled to confirm it is available.
2. REST fallback (if you are not using MCP):
POST https://api.agntdata.dev/map
Header: Authorization: Bearer <YOUR_AGNTDATA_API_KEY>
Required parameters:
- url (string, required) — The base URL of the site to map.
Optional parameters:
- search (string, optional) — Order returned URLs by relevance to this query.
- sitemap (string, optional) — How to use the site's sitemap. Default 'include'.
- includeSubdomains (boolean, optional) — Include URLs on subdomains.
- ignoreQueryParameters (boolean, optional) — Treat URLs that differ only by query string as the same URL.
- limit (integer, optional) — Maximum number of URLs to return. Default 5000, max 100000.
- timeout (integer, optional) — Request timeout in milliseconds.
- location (object, optional) — Geographic proxy location and language settings.
3. COST & KEY. This endpoint is priced per call in credits, and each successful response reports its own cost in its meta.costCents field. Only successful calls are charged.
Get an agntdata API key at https://app.agntdata.dev.
4. TEST. Connect the server (or set the key), discover the tool
(data_web_map), run one minimal call with just the required parameters,
and report back the shape of the JSON response (top-level fields).
Reference (machine-readable variant): https://superagnt.com/docs/apis/web/endpoints/map.md{
"name": "map",
"description": "Returns a list of URLs found on a website, quickly. Useful for discovering the structure of a site before scraping specific pages. Optionally order results by relevance to a `search` term.",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "The base URL of the site to map."
},
"search": {
"type": "string",
"description": "Order returned URLs by relevance to this query."
},
"sitemap": {
"type": "string",
"enum": [
"skip",
"include",
"only"
],
"description": "How to use the site's sitemap. Default 'include'.",
"default": "include"
},
"includeSubdomains": {
"type": "boolean",
"description": "Include URLs on subdomains.",
"default": true
},
"ignoreQueryParameters": {
"type": "boolean",
"description": "Treat URLs that differ only by query string as the same URL.",
"default": true
},
"limit": {
"type": "integer",
"description": "Maximum number of URLs to return. Default 5000, max 100000.",
"minimum": 1,
"maximum": 100000,
"default": 5000
},
"timeout": {
"type": "integer",
"description": "Request timeout in milliseconds.",
"minimum": 1000
},
"location": {
"type": "object",
"description": "Geographic proxy location and language settings.",
"properties": {
"country": {
"type": "string",
"description": "ISO 3166-1 alpha-2 country code."
},
"languages": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
},
"required": [
"url"
]
}
}| Name | In | Type | Required | Description |
|---|---|---|---|---|
url | body | string | required | The base URL of the site to map. |
search | body | string | optional | Order returned URLs by relevance to this query. |
sitemap | body | string | optional | How to use the site's sitemap. Default 'include'. |
includeSubdomains | body | boolean | optional | Include URLs on subdomains. |
ignoreQueryParameters | body | boolean | optional | Treat URLs that differ only by query string as the same URL. |
limit | body | integer | optional | Maximum number of URLs to return. Default 5000, max 100000. |
timeout | body | integer | optional | Request timeout in milliseconds. |
location | body | object | optional | Geographic proxy location and language settings. |
The list of URLs discovered on the site.
{
"type": "object"
}This endpoint is priced per call and deducted from your agntdata balance; only a successful call is billed. Every billable response reports its own cost in meta.costCents.
Get a key at app.agntdata.dev.
start calling
Point your client at https://mcp.superagnt.com/mcp and your agent has this endpoint, plus every other source on one balance.