# Sitemap parser API

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

## When an agent wants this

- Find which pages of a documentation site to read before calling /extract
- Filter URLs by a substring and page through large sitemaps
- Read source-reported last-modified dates

## Sitemap parser, URL extractor: documentation URLs from a bounded XML sitemap

Sitemap parser, URL extractor: documentation URLs from a bounded XML sitemap. $0.002 per successful call; failed calls are never charged.

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

Inputs: url (required): Absolute public HTTP(S) XML sitemap URL; child sitemaps are not fetched automatically; contains: Optional case-insensitive URL substring, at most 200 characters; maxUrls: Reported matching locations examined in one page (1-100, default 25); invalid/nonpublic locations may reduce returned entries; offset: Offset into distinct reported matching locations (0-1000, default 0); use nextOffset to continue.

Free first (3 successful calls per caller, no key):

    curl "https://glyphcall.com/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&trial=1"

With a card key from https://glyphcall.com/credits, in a header or, when your client can only fetch a URL, in the URL:

    curl -H "Authorization: Bearer adt_…" "https://glyphcall.com/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml"
    curl "https://glyphcall.com/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&api_key=adt_…"

Without a key the same URL answers 402 with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://glyphcall.com/mcp) this tool is sitemap.

Example result:

    {
      "url": "https://www.sitemaps.org/sitemap.xml",
      "finalUrl": "https://www.sitemaps.org/sitemap.xml",
      "kind": "urlset",
      "entries": [
        {
          "url": "https://www.sitemaps.org/",
          "lastModified": "2016-11-21"
        },
        {
          "url": "https://www.sitemaps.org/protocol.html",
          "lastModified": "2022-12-15"
        }
      ],
      "reportedEntries": 84,
      "uniqueReportedLocations": 84,
      "missingOrInvalidLocations": 0,
      "duplicateReportedLocations": 0,
      "rejectedLocationsInPage": 0,
      "matchingReportedLocations": 84,
      "contains": "",
      "offset": 0,
      "nextOffset": 2,
      "hasMore": true,
      "warnings": [
        "counts_describe_reported_locations_before_public_url_validation",
        "locations_and_dates_are_source_reported_not_probed"
      ],
      "text": "https://www.sitemaps.org/\nhttps://www.sitemaps.org/protocol.html",
      "format": "text",
      "hashFormat": "text",
      "contentHash": "sha256:3dafd81033420b45ac1326612fc50790c7e9d0be0bb1707aa4ec4f1cfb0c7284",
      "sourceHash": "sha256:5be36d03d5aaeb4f498144c8d319bc651b3063b94f5012bfb554090dfcea2e55",
      "fetchedAt": "2026-11-10T19:47:32.627Z"
    }

Tool page: https://glyphcall.com/tools/sitemap

## Robots.txt check: may this crawler fetch this URL?

Robots.txt check: may this crawler fetch this URL?. $0.001 per successful call; failed calls are never charged.

Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.

Inputs: url (required): The page URL to check (its site's /robots.txt is read); agent: Crawler user agent or product token (default *).

Free first (3 successful calls per caller, no key):

    curl "https://glyphcall.com/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&trial=1"

With a card key from https://glyphcall.com/credits, in a header or, when your client can only fetch a URL, in the URL:

    curl -H "Authorization: Bearer adt_…" "https://glyphcall.com/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch"
    curl "https://glyphcall.com/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&api_key=adt_…"

Without a key the same URL answers 402 with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://glyphcall.com/mcp) this tool is robots.

Example result:

    {
      "url": "https://www.google.com/search",
      "robotsUrl": "https://www.google.com/robots.txt",
      "status": 200,
      "agent": "Googlebot",
      "allowed": false,
      "matchedRule": "Disallow: /search",
      "group": "*",
      "rulesInGroup": 280,
      "sitemaps": [
        "https://www.google.com/sitemap.xml"
      ]
    }

Tool page: https://glyphcall.com/tools/robots

## Questions

Docs for agents: https://glyphcall.com/llms.txt · OpenAPI: https://glyphcall.com/openapi.json · All guides: https://glyphcall.com/guides