io.github.sadri-dridi/agent-crawl-readinessweb-automation

Agent Crawl Readiness

Verified · 2 mo ago

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Install

claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Our take

Checks whether a site is ready for agent access by inspecting robots.txt, sitemap.xml, llms.txt, and the homepage. It is useful for site owners, SEO teams, and agent-platform developers auditing discoverability and crawl permissions. The scope is narrow but clear, and the zero-star profile suggests an early project rather than a mature standard tool.

reviewed by hand · 2026-09-26

Something wrong or dead here? Report it

Verification record

last verified
2 mo ago
github stars
0
last commit
2 mo ago
archived
no
in registry since
2026-09-23

github.com/sadri-dridi/named-mcp-utilities

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 30 days ago

Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.

hand-reviewed117k starschecked 30 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 13 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed90k starschecked 13 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 17 days ago

Playwright Tools for MCP

hand-reviewed38k stars3.5m dl/wkchecked 17 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · 22 days ago

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked 22 days ago

aihawkweb-automation

AIHawk

Verified · yesterday

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked yesterday