dev.workers.pathwren.www/ai-crawler-indexweb-automation

AI Crawler Index

Synced · awaiting check

Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.

Install

claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/c/mcp-registry-official/mcp

Our take

AI Crawler Index is a web-operations tool for identifying AI crawlers by user agent or IP, checking claimed ranges, and helping generate robots.txt rules. It is for site operators, infrastructure teams, or compliance-minded publishers rather than general developers. The function is concrete and plausible, but it is a narrow operational utility and the available metadata does not show broader maturity.

reviewed by hand · 2026-09-01

Something wrong or dead here? Report it

Verification record

last verified
today
in registry since
2026-09-01

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · today

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed112k starschecked today

Scraplingweb-automation

Scrapling MCP Server

Verified · 20 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed74k starschecked 20 days ago

skyvernweb-automation

Skyvern

Verified · 19 days ago

AI-powered browser automation — navigate, click, fill forms, and extract data from any website.

hand-reviewed23k starschecked 19 days ago

apify-mcp-serverweb-automation

apify-mcp-server

Verified · 23 days ago

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡

hand-reviewed3.0k starschecked 23 days ago