io.github.crawlerpeek/crawlerpeeksearch

Crawlerpeek

Verified · today

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.

Install

claude mcp add crawlerpeek -- npx -y [email protected]

Our take

Checks whether named AI crawlers can access a URL, using a read-only interface. It is useful for site owners auditing crawler visibility, though it addresses a narrow question and its low adoption suggests results should be confirmed against current robots.txt behavior and crawler policies.

reviewed by hand · 2026-10-26

Something wrong or dead here? Report it

Verification record

last verified
today
github stars
3
last commit
today
archived
no
license
MIT
npm
crawlerpeek
latest version
0.2.0
in registry since
2026-10-25

github.com/crawlerpeek/crawlerpeek

More in search

browse all →
mcp-server-searchsearch

mcp-server-search

Verified · 43 days ago

MCP server for web search operations

hand-reviewed39k starschecked 43 days ago

repomixsearch

Repomix

Verified · 34 days ago

Pack local or remote codebases into AI-friendly files that LLMs and coding agents can read or search

hand-reviewed28k stars67k dl/wkchecked 34 days ago

screenpipe-mcpsearch

screenpipe

Verified · 35 days ago

Search your local screen recordings, audio transcripts, and computer activity from screenpipe.

hand-reviewed22k stars2.8k dl/wkchecked 35 days ago

xerjsearch

XERJ

Verified · yesterday

The search engine for AI agents: cited answers from your local code, docs, specs and logs.

hand-reviewed15k starschecked yesterday

sui-kapasearch

Sui Knowledge Base (kapa.ai)

Verified · 30 days ago

Semantic search across Sui documentation and knowledge sources, powered by kapa.ai

hand-reviewed7.8k starschecked 30 days ago

firecrawl-mcp-serversearch

Firecrawl MCP Server

Verified · 4 days ago

Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.

hand-reviewed7.7k starschecked 4 days ago