Operated directly by Nekuda, this crawler discovers and indexes publicly available WebMCP tools on websites for search. It collects tool metadata and the websites where tools are available to build a searchable index of web capabilities. Collected website content is not used to train AI models.
Discovery includes reading permitted public pages and resources, and loading websites in remote browsers through Browser Use to discover and verify tools registered at runtime. The bot checks robots.txt before fetching site content. It does not invoke those tools, sign in, submit forms, or make purchases.
User-Agent: nekuda-scanner/1.0 (+https://bot.nekuda.ai).
Crawling and browser discovery requests use Ed25519 HTTP Message Signatures (Web Bot Auth). Signature-Agent: "https://bot.nekuda.ai". The public key directory serves a signed proof of control, refreshed automatically. Publishing this identity does not itself imply approval by Cloudflare or permission from any website.
The scanner checks robots.txt before fetching site content and honors applicable Disallow, Crawl-delay, and Request-rate directives. It uses configurable request budgets, respects Retry-After, and pauses or defers work when throttled. Browser verification also checks resource origins, shares request budgets between sessions using the same observed egress IP, and stops on access challenges. There is no assumption that a single request rate is permitted on every website.
Website operators can disallow the crawler in robots.txt:
User-agent: nekuda-scanner Disallow: /
For opt-out requests, crawling concerns, or identity questions, contact founders@nekuda.ai.