io.github.Nero-Engine/ai-crawler-access-checkersearch

AI Crawler Access Checker

Synced · awaiting check

Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

Install

claude mcp add --transport http ai-crawler-access-checker https://mcp.apify.com/?tools=nerolabs/ai-crawler-access-checker

Requires: Authorization

Our take

Checks whether common AI crawlers are permitted to access a supplied list of websites. It is useful for site owners, SEO teams, and compliance reviewers, but the remote-only listing provides no public adoption or maintenance signals, so results should be verified against current robots.txt and HTTP behavior.

reviewed by hand · 2026-11-02

Something wrong or dead here? Report it

Verification record

last verified: today
in registry since: 2026-11-01

More in search

mcp-server-searchsearch

mcp-server-search

Verified · 2 mo ago

MCP server for web search operations

hand-reviewed39k starschecked 2 mo ago

repomixsearch

Repomix

Verified · 41 days ago

Pack local or remote codebases into AI-friendly files that LLMs and coding agents can read or search

hand-reviewed28k stars67k dl/wkchecked 41 days ago

screenpipe-mcpsearch

screenpipe

Verified · 42 days ago

Search your local screen recordings, audio transcripts, and computer activity from screenpipe.

hand-reviewed22k stars2.8k dl/wkchecked 42 days ago

XERJ

Verified · 8 days ago

The search engine for AI agents: cited answers from your local code, docs, specs and logs.

hand-reviewed15k starschecked 8 days ago

sui-kapasearch

Sui Knowledge Base (kapa.ai)

Verified · 37 days ago

Semantic search across Sui documentation and knowledge sources, powered by kapa.ai

hand-reviewed7.8k starschecked 37 days ago

firecrawl-mcp-serversearch

Firecrawl MCP Server

Verified · 11 days ago

Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.

hand-reviewed7.7k starschecked 11 days ago