io.github.taylorsmithgg/crawl-censusweb-automation

Crawl Census

Verified · 11 days ago

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Install

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Our take

Crawl Census is aimed at answering a practical web-automation question: whether a domain is likely to allow, block, or monetize crawler access. It is best suited to teams doing large-scale scraping, indexing, or agent browsing who want a policy check before making requests. The use case is narrow but credible; with no visible adoption or packaging signals, it reads as a specialized, still-maturing utility.

reviewed by hand · 2026-08-24

Something wrong or dead here? Report it

Verification record

last verified
11 days ago
github stars
0
last commit
2 mo ago
archived
no
license
NOASSERTION
in registry since
2026-08-23

github.com/taylorsmithgg/crawl-census-client

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 37 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed112k starschecked 37 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 30 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed79k starschecked 30 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 32 days ago

Playwright Tools for MCP

hand-reviewed37k stars5.9m dl/wkchecked 32 days ago

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 14 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 14 days ago

aihawkweb-automation

AIHawk

Verified · 25 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed30k starschecked 25 days ago