create-custom-grader is a skill in NVIDIA/SkillEvaluator, last changed 17 Nov 2026. It can run shell commands and bundle scripts.
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
At a glance
What it can touch
How much a skill can do, not whether it is risky.
Pattern check
Nothing flagged
That isn't proof the skill is harmless.
Commit 4327960 · 20 Nov 2026
How the check worksPattern check: nothing flagged
That isn't proof the skill is harmless.
QAI Index
We rate every listed skill the same way, from 0 to 100.
A guide to compare by, not a verdict on the skill.
How it's scoredGet an email when its check changes.
Keep it with the other skills you use, installed with one command.
Install
Sign in to see how to install it
Free with Google or an email link, no password needed. Then pick your agent and copy its steps.
What it can touch
Sign in to see the file and line behind each one
The summary at the top stays open to everyone. Signing in is free, with Google or an email link.
Pattern check
Pattern check: nothing flagged · commit 4327960 · 20 Nov 2026
We checked every text file in this skill, at commit 4327960 on 20 Nov 2026, against 45 published patterns such as download-and-run commands, hidden text and credential access. We don't run the skill, we don't read the web pages it points to, and we can't judge everything its instructions might lead an agent to do. In published tests, pattern checks like this one missed most malicious skills. Read the skill before you install it — especially anything it asks you to download or run.
Format check
Each agent reads SKILL.md a little differently. We check the file against each one's published rules; we don't run the skill in any agent.
Agent Skills specPasses
The format our Agent Skills docs set out, held to the strictest version of each rule among the agents we check.
- Note
Anthropic's packaging script leaves these out: evals/. F8
Fix: Delete them, or list them in .gitignore.
- Note
There is no license field, so readers can't tell whether they may reuse the skill. O2
Fix: Add license: with an SPDX name such as MIT or Apache-2.0, or the name of a bundled licence file.
- Note
Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4
Fix: Link each one from SKILL.md and say when to read it.
- Note
Claude CodePasses
Claude Code reads its own extra fields, ignores unknown ones, and still loads a skill whose frontmatter is broken, without its fields.
- Note
Anthropic's packaging script leaves these out: evals/. F8
Fix: Delete them, or list them in .gitignore.
- Note
Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4
Fix: Link each one from SKILL.md and say when to read it.
- Note
Claude apps & API uploadPasses
Uploading to claude.ai or the Claude API, or packaging with Anthropic's script, accepts only the six spec fields and stops on anything else.
- Note
Anthropic's packaging script leaves these out: evals/. F8
Fix: Delete them, or list them in .gitignore.
- Note
Reference files that SKILL.md doesn't link: evals/files/create-custom-grader-001/benchmark/prompts/direct.md, evals/files/create-custom-grader-001/benchmark/prompts/vague.md and evals/files/create-custom-grader-001/benchmark/rubric.md. Anthropic suggests linking every reference file directly from SKILL.md. B4
Fix: Link each one from SKILL.md and say when to read it.
- Note
OpenAI (Codex & API)Passes
Codex finds SKILL.md by its exact name and needs a description; OpenAI's Skills API checks the frontmatter against the spec and caps uploads.
- Note
Anthropic's packaging script leaves these out: evals/. F8
Fix: Delete them, or list them in .gitignore.
- Note
VS Code & CopilotPasses
VS Code reads skills one folder below .github/skills, .claude/skills or .agents/skills and falls back to the folder name when the name is missing or differs; GitHub's Copilot docs require the exact name SKILL.md.
- Note
Anthropic's packaging script leaves these out: evals/. F8
Fix: Delete them, or list them in .gitignore.
- Note
SKILL.md
Files
11 files
Changelog
Newest first: each commit that changed its folder on GitHub, and each change to its listing here.
- Listed on QAI
- Updated: fix(create-custom-grader): ship the fixtures its eval cases ask for (… (f404bbc (opens in a new tab))
- Updated: feat(datasets): add lossless preview-first dataset migration (#435) (ed1f002 (opens in a new tab))
- Updated: fix(tier3): isolate per-task staged inputs (fe1ea34 (opens in a new tab))
- Updated: docs: prepare SkillEvaluator documentation for launch (fdf3ea4 (opens in a new tab))
- Updated: Initial SkillEvaluator repository snapshot (6a1bc57 (opens in a new tab))
QAI Index badge
Show this skill's QAI Index in a README or on a website. It links back to this page and changes when we refresh the catalog.
[](https://qai.io/skills/nvidia/skillevaluator/create-custom-grader)More from NVIDIA/SkillEvaluator
Other AI & ML engineering skills
Same category, most repo stars first.