io.github.manchittlab/thecrawlersearch

TheCrawler

Verified · 18 days ago

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Install

claude mcp add thecrawler -- npx -y [email protected]

Our take

AGPL-3.0 universal web scraper producing LLM-ready markdown with RAG chunking, PDF/DOCX support, JSON-LD/microdata extraction and adaptive Cheerio->Playwright rendering, exposed via CLI, REST and MCP. The org is new (Apr 2026, 0 stars) and holds several 0-commits-ahead awesome-* forks — but every fork is unmodified (no self-insertion), so it clears the fork-farm test; the dist code is substantial and real. Capable scraper; young project with no traction yet.

reviewed by hand · 2026-07-06

Something wrong or dead here? Report it

Verification record

last verified
18 days ago
github stars
2
last commit
3 mo ago
archived
no
license
AGPL-3.0
downloads / wk
36
latest version
0.5.8
in registry since
2026-07-02

github.com/manchittlab/TheCrawler

mcp-server-searchsearch

mcp-server-search

Verified · 21 days ago

MCP server for web search operations

hand-reviewed39k starschecked 21 days ago

repomixsearch

Repomix

Verified · 12 days ago

Pack local or remote codebases into AI-friendly files that LLMs and coding agents can read or search

hand-reviewed28k stars67k dl/wkchecked 12 days ago

screenpipe-mcpsearch

screenpipe

Verified · 14 days ago

Search your local screen recordings, audio transcripts, and computer activity from screenpipe.

hand-reviewed22k stars2.8k dl/wkchecked 14 days ago

sui-kapasearch

Sui Knowledge Base (kapa.ai)

Verified · 8 days ago

Semantic search across Sui documentation and knowledge sources, powered by kapa.ai

hand-reviewed7.8k starschecked 8 days ago

firecrawl-mcp-serversearch

Firecrawl MCP Server

Verified · 28 days ago

MCP server for Firecrawl — web search, scraping, and biomedical/arXiv paper search.

hand-reviewed7.4k starschecked 28 days ago

robinsearch

Robin: AI-Powered Dark Web OSINT

Verified · 8 days ago

Dark web OSINT over Tor: search onion engines, scrape pages, report with your own model.

hand-reviewed7.3k starschecked 8 days ago