io.github.sharozdawa/robotstxt-aiweb-automation

robotstxt-ai

Verified · 22 days ago

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Install

claude mcp add robotstxt-ai -- npx -y [email protected]

Our take

This server analyzes and generates `robots.txt` rules with explicit attention to AI crawlers such as GPTBot and ClaudeBot. It is for site owners, SEO practitioners, and developers managing crawler policy as AI bot traffic becomes more relevant. The use case is timely and concrete, though users should still validate generated policies carefully because crawler behavior and preferred directives can change faster than a small utility may be maintained.

reviewed by hand · 2026-07-28

Something wrong or dead here? Report it

Verification record

last verified
22 days ago
github stars
0
last commit
5 mo ago
archived
no
license
MIT
npm
robotstxt-ai-mcp
latest version
1.0.1
in registry since
2026-03-21

github.com/sharozdawa/robotstxt-ai

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 14 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed108k starschecked 14 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 8 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed74k starschecked 8 days ago

skyvernweb-automation

Skyvern

Verified · 7 days ago

AI-powered browser automation — navigate, click, fill forms, and extract data from any website.

hand-reviewed23k starschecked 7 days ago

apify-mcp-serverweb-automation

apify-mcp-server

Verified · 11 days ago

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡

hand-reviewed3.0k starschecked 11 days ago