dev.workers.mike-tusa.robotsgate/robotsgateweb-automation

RobotsGate

Synced · awaiting check

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Install

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Requires: Authorization

Our take

Generates and validates robots.txt rules, with specific support for checking access by AI crawlers. It suits site operators and web-automation agents, though the remote-only service and optional paid tier warrant checking rule accuracy and service availability before relying on it for production controls.

reviewed by hand · 2026-10-17

Something wrong or dead here? Report it

Verification record

last verified
today
in registry since
2026-10-16

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 8 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed117k starschecked 8 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 39 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed79k starschecked 39 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 41 days ago

Playwright Tools for MCP

hand-reviewed37k stars5.9m dl/wkchecked 41 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · today

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked today

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 23 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 23 days ago