io.github.nolindnaidoo/scrape-leweb-automation

Scrape-LE

Verified · 19 days ago

Analyse robots.txt content and report whether a path may be crawled.

Install

claude mcp add scrape-le -- npx -y [email protected]

Our take

Scrape-LE checks `robots.txt` rules and reports whether a given path appears crawlable. It is aimed at agents or automation workflows that need a lightweight compliance check before scraping. The scope is narrow and the public signal is minimal, so it reads as a small utility rather than a broadly proven crawling tool.

reviewed by hand · 2026-08-11

Something wrong or dead here? Report it

Verification record

last verified
19 days ago
github stars
1
last commit
26 days ago
archived
no
license
MIT
npm
scrape-le-mcp
latest version
2.2.5
in registry since
2026-08-05

github.com/nolindnaidoo/scrape-le

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 19 days ago

Control a real Chrome browser to complete any task: fill forms, extract data, book flights.

hand-reviewed112k starschecked 19 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 12 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed79k starschecked 12 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 14 days ago

Playwright Tools for MCP

hand-reviewed37k stars5.9m dl/wkchecked 14 days ago

aihawkweb-automation

AIHawk

Verified · 7 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed30k starschecked 7 days ago

skyvernweb-automation

Skyvern

Verified · 9 days ago

AI-powered browser automation — navigate, click, fill forms, and extract data from any website.

hand-reviewed23k starschecked 9 days ago