io.github.nolindnaidoo/scrape-leweb-automation

Scrape-LE

Verified · 27 days ago

Analyse robots.txt content and report whether a path may be crawled.

Install

claude mcp add scrape-le -- npx -y [email protected]

Our take

Scrape-LE checks `robots.txt` rules and reports whether a given path appears crawlable. It is aimed at agents or automation workflows that need a lightweight compliance check before scraping. The scope is narrow and the public signal is minimal, so it reads as a small utility rather than a broadly proven crawling tool.

reviewed by hand · 2026-08-11

Something wrong or dead here? Report it

Verification record

last verified
27 days ago
github stars
1
last commit
27 days ago
archived
no
license
MIT
npm
scrape-le-mcp
latest version
2.4.0
in registry since
2026-08-05

github.com/nolindnaidoo/scrape-le

More in web-automation

browse all →
browser-useweb-automation

browser-use

Verified · 26 days ago

Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.

hand-reviewed117k starschecked 26 days ago

Scraplingweb-automation

Scrapling MCP Server

Verified · 9 days ago

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

hand-reviewed90k starschecked 9 days ago

playwright-mcpweb-automation

playwright-mcp

Verified · 13 days ago

Playwright Tools for MCP

hand-reviewed38k stars3.5m dl/wkchecked 13 days ago

next-ai-drawioweb-automation

Next AI Draw.io

Verified · 18 days ago

draw.io diagrams your AI agent draws on a live browser canvas that you can also edit by hand

hand-reviewed36k stars16k dl/wkchecked 18 days ago

invisible-playwright-mcpweb-automation

invisible_playwright_mcp

Verified · 41 days ago

AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.

hand-reviewed32k starschecked 41 days ago