io.github.manchittlab/thecrawlersearch

TheCrawler

Verified · today

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Install

claude mcp add thecrawler -- npx -y [email protected]

Our take

AGPL-3.0 universal web scraper producing LLM-ready markdown with RAG chunking, PDF/DOCX support, JSON-LD/microdata extraction and adaptive Cheerio->Playwright rendering, exposed via CLI, REST and MCP. The org is new (Apr 2026, 0 stars) and holds several 0-commits-ahead awesome-* forks — but every fork is unmodified (no self-insertion), so it clears the fork-farm test; the dist code is substantial and real. Capable scraper; young project with no traction yet.

reviewed by hand · 2026-07-06

Something wrong or dead here? Report it

Verification record

last verified: today
github stars: 0
last commit: 4 days ago
archived: no
license: AGPL-3.0
npm: thecrawler
downloads / wk: 586
latest version: 0.5.8
in registry since: 2026-07-02

github.com/manchittlab/TheCrawler