create-custom-grader

NVIDIA/SkillEvaluatorAI & ML engineering

create-custom-grader is a skill in NVIDIA/SkillEvaluator, last changed 17 Nov 2026. It can run shell commands and bundle scripts.

From the author
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.

At a glance

What it can touch

How much a skill can do, not whether it is risky.

Pattern check

Nothing flagged

That isn't proof the skill is harmless.

Commit 4327960 · 20 Nov 2026

How the check works

Pattern check: nothing flagged

That isn't proof the skill is harmless.

QAI Index

77/QI

We rate every listed skill the same way, from 0 to 100.

A guide to compare by, not a verdict on the skill.

How it's scored

How the QAI Index is scored

Adoption · up to 40
GitHub stars on the repo that holds the skill, shared among its skills, and copies of the skill in other repos.
Quality · up to 30
Our pattern check, the format check, a named licence and a description of a sentence or more.
Upkeep · up to 20
How recently the skill's folder changed.
Trust · up to 10
An official collection, or an organisation that has confirmed its domain with GitHub.

Get an email when its check changes.

Keep it with the other skills you use, installed with one command.

Install

Sign in to see how to install it

Free with Google or an email link, no password needed. Then pick your agent and copy its steps.

Apache 2.0
Licence
4.3k
Repo stars
Publisher

What it can touch

Sign in to see the file and line behind each one

The summary at the top stays open to everyone. Signing in is free, with Google or an email link.

Pattern check

Pattern check: nothing flagged · commit 4327960 · 20 Nov 2026

We checked every text file in this skill, at commit 4327960 on 20 Nov 2026, against 45 published patterns such as download-and-run commands, hidden text and credential access. We don't run the skill, we don't read the web pages it points to, and we can't judge everything its instructions might lead an agent to do. In published tests, pattern checks like this one missed most malicious skills. Read the skill before you install it — especially anything it asks you to download or run.

How the check worksReport an incorrect finding

Format check

Each agent reads SKILL.md a little differently. We check the file against each one's published rules; we don't run the skill in any agent.

  • Agent Skills specPasses

    The format our Agent Skills docs set out, held to the strictest version of each rule among the agents we check.

    • Note

      Anthropic's packaging script leaves these out: evals/. F8

      Fix: Delete them, or list them in .gitignore.

    • Note

      There is no license field, so readers can't tell whether they may reuse the skill. O2

      Fix: Add license: with an SPDX name such as MIT or Apache-2.0, or the name of a bundled licence file.

    • Note

      Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4

      Fix: Link each one from SKILL.md and say when to read it.

  • Claude CodePasses

    Claude Code reads its own extra fields, ignores unknown ones, and still loads a skill whose frontmatter is broken, without its fields.

    • Note

      Anthropic's packaging script leaves these out: evals/. F8

      Fix: Delete them, or list them in .gitignore.

    • Note

      Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4

      Fix: Link each one from SKILL.md and say when to read it.

  • Claude apps & API uploadPasses

    Uploading to claude.ai or the Claude API, or packaging with Anthropic's script, accepts only the six spec fields and stops on anything else.

    • Note

      Anthropic's packaging script leaves these out: evals/. F8

      Fix: Delete them, or list them in .gitignore.

    • Note

      Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4

      Fix: Link each one from SKILL.md and say when to read it.

  • OpenAI (Codex & API)Passes

    Codex finds SKILL.md by its exact name and needs a description; OpenAI's Skills API checks the frontmatter against the spec and caps uploads.

    • Note

      Anthropic's packaging script leaves these out: evals/. F8

      Fix: Delete them, or list them in .gitignore.

  • VS Code & CopilotPasses

    VS Code reads skills one folder below .github/skills, .claude/skills or .agents/skills and falls back to the folder name when the name is missing or differs; GitHub's Copilot docs require the exact name SKILL.md.

    • Note

      Anthropic's packaging script leaves these out: evals/. F8

      Fix: Delete them, or list them in .gitignore.

SKILL.md

Files

Changelog

Newest first: each commit that changed its folder on GitHub, and each change to its listing here.

  1. Listed on QAI
  2. Updated: fix(create-custom-grader): ship the fixtures its eval cases ask for (… (f404bbc (opens in a new tab))
  3. Updated: feat(datasets): add lossless preview-first dataset migration (#435) (ed1f002 (opens in a new tab))
  4. Updated: fix(tier3): isolate per-task staged inputs (fe1ea34 (opens in a new tab))
  5. Updated: docs: prepare SkillEvaluator documentation for launch (fdf3ea4 (opens in a new tab))
  6. Updated: Initial SkillEvaluator repository snapshot (6a1bc57 (opens in a new tab))

QAI Index badge

Show this skill's QAI Index in a README or on a website. It links back to this page and changes when we refresh the catalog.

Preview of the QAI Index badge
[![QAI Index](https://qai.io/badge/skills/nvidia/skillevaluator/create-custom-grader.svg)](https://qai.io/skills/nvidia/skillevaluator/create-custom-grader)

More from NVIDIA/SkillEvaluator

SkillQAI IndexWhat it can touchPattern checkLicenceUpdated
api-callerCall any REST API dynamically. Make GET, POST, PUT, DELETE requests to any endpoint with custom headers and JSON body.
78/QI
  • Runs shell commands
  • +3 more

Pattern check: Nothing flaggedApache 2.0Updated 19 Nov 2026

78/QI
  • Runs shell commands
  • Reaches the network
  • +2 more
Pattern check: Nothing flaggedApache 2.019 Nov 2026
78/QI
  • Runs shell commands
  • Bundles scripts
Pattern check: Nothing flaggedApache 2.020 Nov 2026
72/QI
  • Installs more skills
  • Runs shell commands
  • +2 more
Pattern check: Nothing flaggedApache 2.017 Nov 2026
task-listRequired for 4+ step requests; add tasks at start and update status after each step.
78/QI
  • Bundles scripts

Pattern check: Nothing flaggedApache 2.0Updated 19 Nov 2026

78/QI
  • Bundles scripts
Pattern check: Nothing flaggedApache 2.019 Nov 2026
77/QI
  • Runs shell commands
  • Bundles scripts
Pattern check: Nothing flaggedApache 2.017 Nov 2026

Other AI & ML engineering skills

Same category, most repo stars first.

SkillQAI IndexWhat it can touchPattern checkLicenceUpdated
68/QI
  • Runs shell commands
Pattern check: Nothing flaggedMIT11 Aug 2026
68/QI
  • Runs shell commands
  • Names web addresses
Pattern check: Nothing flaggedMIT11 Aug 2026
claude-apiReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).

anthropics/skills

72/QI
  • Touches credentials
  • +6 more

Pattern check: 1 item to reviewApache 2.0Updated 9 Oct 2026

72/QI
  • Touches credentials
  • Downloads software
  • +5 more
Pattern check: 1 item to reviewApache 2.09 Oct 2026
88/QI
  • Touches credentials
  • Runs shell commands
  • +3 more
Pattern check: Nothing flaggedApache 2.028 Oct 2026
benchmark-modelsCross-model benchmark for gstack skills. Use for: cross model benchmark, compare claude gpt gemini. (gstack)

garrytan/gstack

70/QI
  • Touches credentials
  • +3 more

Pattern check: 2 items to reviewMITUpdated 19 Nov 2026

70/QI
  • Touches credentials
  • Runs shell commands
  • +2 more
Pattern check: 2 items to reviewMIT19 Nov 2026
70/QI
  • Touches credentials
  • Runs shell commands
  • +2 more
Pattern check: 3 items to reviewMIT20 Nov 2026

Repo facts from GitHub on 20 Nov 2026. Files read at commit 4327960.

QAI did not write these skills and is not affiliated with their authors unless a page says so. Logos and icons belong to their owners and are not a mark of approval. How we check