io.github.Grubbomatic/crawl-readinessweb-automation

crawl-readiness

Verified · 2 mo ago

Check what AI crawlers may read and what a page permits, generate llms.txt and schema.

Install

claude mcp add crawl-readiness -- npx -y [email protected]

Requires: CRAWL_READINESS_API_KEY, CRAWL_READINESS_API_BASE

Our take

Audits websites for AI crawler readiness and can generate llms.txt, robots.txt, schema, and AI-mention tracking outputs. It is for SEO, marketing, and web teams adapting sites for AI discovery; the feature set is broad, so users should verify the generated policy files and crawler assumptions before publishing them.

reviewed by hand · 2026-09-21

Something wrong or dead here? Report it

Verification record

last verified
2 mo ago
github stars
0
last commit
2 mo ago
archived
no
license
MIT
npm
crawl-readiness-mcp
downloads / wk
207
latest version
0.2.3
in registry since
2026-09-20

github.com/Grubbomatic/crawl-readiness-mcp

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 27 days ago

Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.

hand-reviewed117k starschecked 27 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 10 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed90k starschecked 10 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 14 days ago

Playwright Tools for MCP

hand-reviewed38k stars3.5m dl/wkchecked 14 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · 19 days ago

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked 19 days ago

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 42 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 42 days ago