io.github.razilzul/llm-eval-harnessai

LLM Eval Harness

Verified · today

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Install

claude mcp add llm-eval-harness -- docker run -i --rm ghcr.io/razilzul/llm-eval-harness:1.0.0

Requires: ANTHROPIC_API_KEY, GEMINI_API_KEY

Our take

Runs golden-set evaluations and compares accuracy, latency, and per-request cost before model changes are accepted. It is useful for engineering teams managing LLM upgrades, though the zero-star OCI project appears early and should be checked for metric definitions, repeatability, and CI integration before relying on its gates.

reviewed by hand · 2026-10-19

Something wrong or dead here? Report it

Verification record

last verified
today
github stars
0
last commit
today
archived
no
license
MIT
in registry since
2026-10-18

github.com/razilzul/llm-eval-harness

More in ai

browse all →
reaai

REA

Verified · 2 days ago

Reverse engineer anything from your terminal or agent with one CLI and MCP server.

hand-reviewed291k stars232k dl/wkchecked 2 days ago

claude-flowai

claude-flow

Verified · 31 days ago

AI orchestration with hive-mind swarms, neural networks, and 87 MCP tools for enterprise dev.

hand-reviewed73k stars19k dl/wkchecked 31 days ago

tldrawai

tldraw

Verified · 32 days ago

Draw and visually collaborate with your agents on tldraw's canvas.

hand-reviewed50k starschecked 32 days ago

codebase-memory-mcpai

Codebase Memory

Verified · 22 days ago

Codebase knowledge graph for AI agents — 162 languages, sub-ms queries, 99% fewer tokens.

hand-reviewed45k stars12k dl/wkchecked 22 days ago

openmetadata-mcpai

OpenMetadata

Verified · 14 days ago

Official OpenMetadata MCP: 21 read and write tools for search, lineage, data quality, governance.

hand-reviewed15k starschecked 14 days ago

mobilerunai

mobilerun

Verified · 34 days ago

Control real Android and iOS devices with LLM agents — tap, swipe, type, automate flows.

hand-reviewed9.4k starschecked 34 days ago