io.github.mark1russell7/polite-crawlerweb-automation

polite-crawler

Verified · today

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Install

claude mcp add polite-crawler -- npx -y [email protected]

Requires: POLITE_CRAWLER_DIR, POLITE_CRAWLER_CONTACT, POLITE_CRAWLER_ALLOW_PRIVATE, POLITE_CRAWLER_ALLOW_USER_PROVIDED

Our take

A web-crawling server for agents that respects robots.txt, avoids repeated fetches, and supports quote discovery and verification. It is aimed at research and citation workflows, and the explicit crawling constraints are a good design signal, though zero stars suggest it is still early or lightly adopted.

reviewed by hand · 2026-10-14

Something wrong or dead here? Report it

Verification record

last verified
today
github stars
0
last commit
today
archived
no
license
MIT
npm
polite-crawler
latest version
0.1.0
in registry since
2026-10-13

github.com/mark1russell7/crawler

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 5 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed117k starschecked 5 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 36 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed79k starschecked 36 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 38 days ago

Playwright Tools for MCP

hand-reviewed37k stars5.9m dl/wkchecked 38 days ago

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 20 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 20 days ago

aihawkweb-automation

AIHawk

Verified · 31 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed30k starschecked 31 days ago