dev.workers.quietworklabs.robots-check/robots-checkweb-automation

Robots.txt & AI Crawler Checker

Synced · awaiting check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Install

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Our take

Tests robots.txt behavior against RFC 9309, checks access rules for named AI crawlers, and inspects sitemap availability. It is a focused utility for site owners, SEO teams, and crawler developers, though the remote-only listing provides no visible adoption or maintenance evidence and results may need confirmation against crawler-specific behavior.

reviewed by hand · 2026-10-19

Something wrong or dead here? Report it

Verification record

last verified
today
in registry since
2026-10-18

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 10 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed117k starschecked 10 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 41 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed79k starschecked 41 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 43 days ago

Playwright Tools for MCP

hand-reviewed37k stars5.9m dl/wkchecked 43 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · 2 days ago

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked 2 days ago

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 25 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 25 days ago