io.github.sharozdawa/robotstxt-aiweb-automation

robotstxt-ai

Stale · 8 mo silent

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Install

claude mcp add robotstxt-ai -- npx -y [email protected]

Our take

This server analyzes and generates `robots.txt` rules with explicit attention to AI crawlers such as GPTBot and ClaudeBot. It is for site owners, SEO practitioners, and developers managing crawler policy as AI bot traffic becomes more relevant. The use case is timely and concrete, though users should still validate generated policies carefully because crawler behavior and preferred directives can change faster than a small utility may be maintained.

reviewed by hand · 2026-07-28

Something wrong or dead here? Report it

Verification record

last verified
2 mo ago
github stars
1
last commit
8 mo ago
archived
no
license
MIT
downloads / wk
25
latest version
1.0.1
in registry since
2026-03-21

github.com/sharozdawa/robotstxt-ai

browser-useweb-automation

browser-use

Verified · 41 days ago

Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.

hand-reviewed117k starschecked 41 days ago

Scrapling MCP Server

Verified · 24 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed90k starschecked 24 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 28 days ago

Playwright Tools for MCP

hand-reviewed38k stars3.5m dl/wkchecked 28 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · 33 days ago

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked 33 days ago

AIHawk

Verified · 12 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 12 days ago