Skills

evaluating-llms-harness

evaluating-llms-harness is a skill in Orchestra-Research/AI-Research-SKILLs, last changed 20 Nov 2025. It can touch credentials, run shell commands, reach the network and name web addresses — read the findings before installing. Our pattern check flagged nothing at commit 773a529 on 27 Oct 2026.

Pattern check: nothing flagged · commit 773a529 · 27 Oct 2026

Install
From the author
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
Licence
MIT
Repo stars
14.0k
Folder last changed
20 Nov 2025 (opens in a new tab)
Publisher
Orchestra-Research
Files
5 files
Checked
773a529 (opens in a new tab) · 27 Oct 2026

Install

Each tab says whether it installs commit 773a529, the one we checked, or the latest version.

No tab for your agent? Use Any agent for OpenCode, Goose, Windsurf (Devin Desktop), Zed, Roo Code, Factory Droid, Junie, Amp and Antigravity, and Claude Code for Cline. Kiro needs the folder copied by hand.

Pinned to commit 773a529, the one we checked.

From your project's root folder:

tmp="$(mktemp -d)" &&
  git clone -q --depth 1 --filter=blob:none --no-checkout https://github.com/Orchestra-Research/AI-Research-SKILLs.git "$tmp" &&
  git -C "$tmp" fetch -q --depth 1 origin 773a52944ba4747a18bd4ae9ade53fff041adcbc &&
  git -C "$tmp" sparse-checkout set -- 11-evaluation/lm-evaluation-harness &&
  git -C "$tmp" checkout -q 773a52944ba4747a18bd4ae9ade53fff041adcbc &&
  mkdir -p .claude/skills &&
  mkdir .claude/skills/evaluating-llms-harness &&
  cp -R "$tmp"/11-evaluation/lm-evaluation-harness/. .claude/skills/evaluating-llms-harness/ &&
  rm -rf "$tmp"

If Claude Code was already running and this created the skills folder, run /reload-skills.

Claude Code docs (opens in a new tab)

What it can touch

From the files and frontmatter at commit 773a529. This is how much a skill can do, not whether it is risky.

What it can touch: Shell, network or credentials

  • Runs shell commands

    Its instructions or files give the agent commands to run on your computer.

  • Reaches the network

    It sends or fetches data over the internet, for example with curl, an HTTP library or a package install.

  • Names web addresses

    It mentions outside web addresses the agent may visit. We list them but don't follow them.

  • Touches credentials

    It reads, asks for or contains keys, tokens, passwords or similar secrets.

    references/api-evaluation.md:417 · references/api-evaluation.md:418

Pattern check

Pattern check: nothing flagged · commit 773a529 · 27 Oct 2026

We checked every text file in this skill, at commit 773a529 on 27 Oct 2026, against 45 published patterns such as download-and-run commands, hidden text and credential access. We don't run the skill, we don't read the web pages it points to, and we can't judge everything its instructions might lead an agent to do. In published tests, pattern checks like this one missed most malicious skills. Read the skill before you install it — especially anything it asks you to download or run.

How the check worksReport an incorrect finding

Outside links we did not follow

  • https://github.com/EleutherAI/lm-evaluation-harness
  • https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs
  • https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
  • https://api.openai.com/v1
  • https://api.anthropic.com
  • https://platform.openai.com/docs/api-reference
  • https://docs.anthropic.com/claude/reference
  • https://huggingface.co/docs/accelerate/
  • https://docs.vllm.ai/
  • https://docs.nvidia.com/nemo-framework/

Format check

Each agent reads SKILL.md a little differently. We check the file against each one's published rules; we don't run the skill in any agent.

  • Agent Skills specFails

    The open format from agentskills.io, held to what the strictest agents enforce, so one folder loads everywhere.

    • Error

      The name "evaluating-llms-harness" doesn't match its folder "lm-evaluation-harness". The spec requires them to match. N4

      Fix: Set name: lm-evaluation-harness, or rename the folder to evaluating-llms-harness.

    • Warning

      The frontmatter uses [ ] or { } values. Most agents accept them, but the spec's validator (skills-ref) rejects the file. F4

      Fix: Write lists as "- item" lines and maps as indented key: value lines, without anchors or tags.

    • Warning

      Fields outside the spec: version, author, tags and dependencies. Uploads to the Claude apps and API, and the spec's validator, reject the file; most agents ignore these fields. O1

      Fix: Delete them for a portable skill, or keep them if you only target the agents that read them.

    • Note

      Reference files over 100 lines with no table of contents at the top: references/api-evaluation.md (491 lines), references/benchmark-guide.md (489 lines), references/custom-tasks.md (603 lines) and references/distributed-eval.md (520 lines). B5

      Fix: Add a short list of the sections at the top of each.

  • Claude CodePasses

    Claude Code reads its own extra fields, ignores unknown ones, and still loads a skill whose frontmatter is broken, without its fields.

    • Note

      The name "evaluating-llms-harness" doesn't match its folder "lm-evaluation-harness". Claude Code shows the skill as /evaluating-llms-harness, and the folder name also invokes it. N4

      Fix: Set name: lm-evaluation-harness, or rename the folder to evaluating-llms-harness.

    • Note

      Claude Code ignores fields it doesn't know: version, author, tags and dependencies. O1

      Fix: Check the spelling against Claude Code's field list, or delete them.

    • Note

      Reference files over 100 lines with no table of contents at the top: references/api-evaluation.md (491 lines), references/benchmark-guide.md (489 lines), references/custom-tasks.md (603 lines) and references/distributed-eval.md (520 lines). B5

      Fix: Add a short list of the sections at the top of each.

  • Claude apps & API uploadFails

    Uploading to claude.ai or the Claude API, or packaging with Anthropic's script, accepts only the six spec fields and stops on anything else.

    • Error

      The name "evaluating-llms-harness" doesn't match its folder "lm-evaluation-harness". claude.ai refuses an upload whose folder name doesn't match the skill name. N4

      Fix: Set name: lm-evaluation-harness, or rename the folder to evaluating-llms-harness.

    • Error

      Upload and packaging fail on fields outside the spec: version, author, tags and dependencies. O1

      Fix: Delete them, and keep your own data under metadata as text values.

    • Note

      The description is 309 characters. The claude.ai Help Center says 200 at most, though Anthropic's API docs and validator allow 1,024. D4

      Fix: If the claude.ai upload refuses it, trim the description to 200 characters.

    • Note

      Reference files over 100 lines with no table of contents at the top: references/api-evaluation.md (491 lines), references/benchmark-guide.md (489 lines), references/custom-tasks.md (603 lines) and references/distributed-eval.md (520 lines). B5

      Fix: Add a short list of the sections at the top of each.

  • OpenAI (Codex & API)Passes with warnings

    Codex finds SKILL.md by its exact name and needs a description; OpenAI's Skills API checks the frontmatter against the spec and caps uploads.

    • Warning

      The name "evaluating-llms-harness" doesn't match its folder "lm-evaluation-harness". OpenAI's own upload example uses a differently named folder, but the spec it follows says they must match. N4

      Fix: Set name: lm-evaluation-harness, or rename the folder to evaluating-llms-harness.

    • Warning

      Fields outside the spec: version, author, tags and dependencies. Codex ignores them; OpenAI doesn't say whether its upload accepts them. O1

      Fix: Delete them, or move your own data under metadata as text values.

  • VS Code & CopilotPasses with warnings

    VS Code reads skills one folder below .github/skills, .claude/skills or .agents/skills and falls back to the folder name when the name is missing or differs; GitHub's Copilot docs require the exact name SKILL.md.

    • Warning

      The name "evaluating-llms-harness" doesn't match its folder "lm-evaluation-harness". VS Code uses the folder name, lm-evaluation-harness, instead. N4

      Fix: Set name: lm-evaluation-harness, or rename the folder to evaluating-llms-harness.

    • Note

      VS Code ignores fields it doesn't support: version, author, tags and dependencies. O1

      Fix: Delete them unless another agent you target reads them.

SKILL.md

Files

5 files

PathSize
SKILL.md (opens in a new tab)11.6 KB
references/api-evaluation.md (opens in a new tab)10.9 KB
references/benchmark-guide.md (opens in a new tab)10.5 KB
references/custom-tasks.md (opens in a new tab)12.8 KB
references/distributed-eval.md (opens in a new tab)11.2 KB

Copies

The same folder appears in 1 other repo. We list it once.

More from Orchestra-Research/AI-Research-SKILLs

  • academic-plotting

    Orchestra-Research/AI-Research-SKILLs

    Generates publication-quality figures for ML papers from research context. Given a paper section or description, extracts system components and relationships to generate architecture diagrams via Gemini. Given experiment results or data, auto-selects chart type and generates data-driven figures via matplotlib/seaborn. Use when creating any figure for a conference paper.

    • Names web addresses
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 25 Mar 2026

  • ara-compiler

    Orchestra-Research/AI-Research-SKILLs

    Compiles any research input — PDF papers, GitHub repositories, experiment logs, code directories, or raw notes — into a complete Agent-Native Research Artifact (ARA) with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph, and grounded evidence. Use when ingesting a paper or codebase into a structured, machine-executable knowledge package, building an ARA from scratch, or converting research outputs into a falsifiable, agent-traversable form.

    • Instructions only
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 27 Apr 2026

  • ara-research-manager

    Orchestra-Research/AI-Research-SKILLs

    Records research provenance as a post-task epilogue, scanning conversation history at the end of a coding or research session to extract decisions, experiments, dead ends, claims, heuristics, and pivots, and writing them into the ara/ directory with user-vs-AI provenance tags. Use as a session epilogue — never during execution — to maintain a faithful, auditable trace of how a research project actually evolved.

    • Runs shell commands
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 27 Apr 2026

  • ara-rigor-reviewer

    Orchestra-Research/AI-Research-SKILLs

    Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release.

    • Instructions only
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 27 Apr 2026

  • audiocraft-audio-generation

    Orchestra-Research/AI-Research-SKILLs

    PyTorch library for audio generation including text-to-music (MusicGen) and text-to-sound (AudioGen). Use when you need to generate music from text descriptions, create sound effects, or perform melody-conditioned music generation.

    • Downloads software
    • Runs shell commands
    • +2 more
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 25 Nov 2025

  • autogpt-agents

    Orchestra-Research/AI-Research-SKILLs

    Autonomous AI agent platform for building and deploying continuous agents. Use when creating visual workflow agents, deploying persistent autonomous agents, or building complex multi-step AI automation systems.

    • Touches credentials
    • Runs shell commands
    • +2 more
    Pattern check: Nothing flagged

    MIT · 14.0k repo stars · Updated 25 Nov 2025

All skills in Orchestra-Research/AI-Research-SKILLs