Audit Agent Skills With a Practical Readiness Scorecard

A credible Agent Skills audit separates four questions: is the skill valid, can the target runtime discover it, can an agent execute it safely, and has the behavior been observed? The scorecard below gives one point to ten evidence-backed criteria. It does not treat our persona layout, trigger headings, or .github/ convention as requirements of the open standard.

Why Most Skill Audits Mix Unlike Things

A repository can contain valid SKILL.md files and still perform poorly. It can also contain excellent procedures in a directory the target runtime never discovers.

That creates four different audit dimensions:

The Agent Skills specification governs the first dimension. Vendor documentation governs installation and discovery. The remaining quality judgments are engineering practice.

Before Scoring: Declare the Audit Target

Write down:

• repository and revision;
• runtime and product surface;
• runtime version, if available;
• operating system or hosted environment;
• skill directories expected to load;
• whether activation is explicit or implicit;
• actions the evaluator is authorized to perform.

Without that scope, “compatible with Copilot, Codex, and Claude” is too broad to verify.

Gate 1: Specification Compliance

Run this gate before assigning quality points.

Required Structure

• A skill is a directory.
• The directory contains SKILL.md.
• The file starts with YAML frontmatter.
• name and description are present and valid.
• The name matches the directory and naming constraints.

Supporting scripts, references, assets, templates, and other files are allowed. Their presence is not a defect.

Use the reference validator linked by the Agent Skills specification, then inspect any warnings manually. If the skill fails required metadata validation, mark it invalid and fix that before interpreting the quality score.

Gate 2: Vendor Discovery

Confirm the claimed runtime supports the installation:

An explicit instruction telling an agent to read .github/skills/foo/SKILL.md may work, but that is not identical to native auto-discovery. Record which mechanism was tested.

If this gate fails, the skill may still be well written; it is simply not installed correctly for the audit target.

The Ten-Point Readiness Score

Award a point only when evidence exists.

1. Valid Skill Contract

The required directory and metadata validate.

Evidence: validator output and inspected frontmatter.

2. Clear Discovery Description

The description states what the skill does and when to use it. It distinguishes important neighboring tasks.

Evidence: positive and boundary prompt review.

3. Focused Outcome

The body defines one coherent capability and an observable result. It may include several steps, but they contribute to that result.

Evidence: output specification or acceptance criteria.

4. Actionable Workflow

The agent can follow ordered steps, decision rules, templates, or scripts without inventing the missing process.

Evidence: dry run or completed artifact.

5. Safety and Authority Boundaries

Destructive changes, external writes, credentials, personal data, provider actions, and approvals are handled explicitly.

Evidence: documented stop conditions plus technical controls where required.

6. Context-Efficient Resources

The entrypoint is readable, references are direct, and large details or deterministic scripts load only when necessary.

Evidence: file graph review and resolved relative links.

7. Proportionate Validation

The skill says what to verify and distinguishes automated checks from browser, provider, production, or human sign-off evidence.

Evidence: runnable commands and truthful evidence labels.

8. Observed Routing and Adherence

Realistic prompts show the expected workflow activating and being followed. Boundary prompts do not activate it incorrectly.

Evidence: recorded runtime, prompt, activation signal, and output.

9. Verified Runtime Compatibility

Every runtime named in the compatibility claim has a documented installation and a tested execution path, or the limitation is stated.

Evidence: per-runtime results—not inferred portability from Markdown alone.

10. Ownership and Freshness

Paths, commands, versions, and sources are current. The skill has an owner or a clear review trigger.

Evidence: repository checks and review history.

Score Bands

These bands are our interpretation, not an industry certification:

A score of 10 is not permanent. A renamed command or vendor path can lower it tomorrow.

Repository Architecture Review

Personas and persistent instructions affect the environment around a skill, but they are not part of the Agent Skills standard. Audit them in a separate section.

Our Convention

In the StartMeUpAI website and ByblosAI application repositories, we generally use:

• AGENTS.md and CLAUDE.md as tool-aware entrypoints;
• .github/agents/ for role briefs;
• .github/instructions/ for path-scoped Copilot rules;
• .github/skills/ as our procedural source of truth;
• technical enforcement through TypeScript, schemas, authorization, linting, tests, and CI.

Review this arrangement for:

• contradictory instructions;
• copied procedures;
• entrypoints that reference missing files;
• path scopes that match nothing;
• critical rules expressed only as model guidance;
• claims of native support that were never tested.

Label findings as repository convention defects, not Agent Skills specification violations.

A Sample Audit Report

Use a report that preserves evidence:

## Audit Target

- Runtime: OpenAI Codex
- Repository revision: <commit>
- Discovery path: .agents/skills
- Activation: implicit and explicit cases

## Compliance

- PASS: Required metadata validates.
- PASS: Supporting references resolve.

## Score

8/10

- Missing point 8: no observed boundary prompt test.
- Missing point 9: Claude compatibility claimed but not tested.

## Required Actions

1. Run three positive and three boundary prompts in Codex.
2. Either test Claude Code or narrow the compatibility claim.

## Evidence Boundary

Format validation passed. Browser and provider behavior were not in scope.

This is more useful than “looks good” because another reviewer can reproduce the result.

How to Test Routing Without Inventing Reasoning

For each skill, create:

• two clear positive prompts;
• two paraphrased positive prompts;
• two boundary prompts;
• one explicit invocation.

Record what activated and what the agent produced. If the runtime does not expose activation, score the observable behavior and note the limitation.

Do not write:

The model compared trigger phrase X with token Y and assigned a 92% match.

Unless a product exposes that exact telemetry, the statement is fictional.

How We Audit Evidence Claims

Our working standard is conservative:

• a passing format validator proves format;
• a passing unit test proves tested code behavior;
• type checking proves type consistency;
• a build proves compilation for the build target;
• a browser run proves only the routes and interactions exercised;
• a provider run proves only the credentials, configuration, and operation exercised;
• human approval must be recorded by a human.

A skill should never convert one class of evidence into another.

Common Audit Mistakes

Awarding Points for a Trigger Heading

A body trigger list can be useful for maintainers and evaluations, but it is not required metadata and normally is not visible during initial discovery.

Penalizing Supporting Files

The standard allows supporting resources. Audit whether they are safe and referenced clearly.

Treating .github/skills as Universal

It is a documented Copilot location and our internal convention. Other runtimes document other native paths.

Scoring File Count

More skills can create more overlap and maintenance. Score capability and evidence, not inventory size.

Claiming Three-Runtime Compatibility After One Test

Format portability is not runtime verification. Test each named product surface.

A Quarterly Review Routine

For skills used in important delivery workflows:

1. Validate all skill directories.
2. Check referenced paths and commands.
3. Review vendor discovery documentation for changes.
4. Re-run positive and boundary routing cases.
5. Exercise safety and approval branches.
6. Review failed or overridden agent runs.
7. Update the score and evidence date.
8. Remove or archive unused skills.

High-change repositories may need event-driven review whenever architecture or tooling changes.

Key Takeaways

1. Separate specification compliance, vendor discovery, execution quality, and maintenance.
2. Pass compliance and discovery gates before interpreting the score.
3. Award points only for inspectable evidence.
4. Treat our personas-and-instructions layout as a convention, not a standard.
5. A high score expires when tools, paths, or workflows change.

Frequently Asked Questions

Is a 10/10 skill guaranteed to work?

No. It means the defined evidence currently covers the ten criteria. New tasks, model behavior, permissions, or repository changes can still reveal defects.

Must every skill have examples?

No. Examples are useful when they reduce ambiguity. This scorecard instead asks whether the workflow is actionable and has been observed.

Can I score a skill before running it?

You can assess compliance and static quality, but you should withhold routing, adherence, and compatibility points until they are observed.

Are personas required for an agent-ready repository?

No. They are part of our preferred architecture when distinct roles add value. A small repository may work well with instructions and a few focused skills.

Official References

• Agent Skills specification and validator
• GitHub Copilot agent skills
• OpenAI Codex customization
• Claude Code skills

Conclusion

A useful audit does more than inspect Markdown. It proves the skill is valid, installed, actionable, safe, observed, and maintainable for a named runtime.

This scorecard reflects how we review engineering evidence: never award a point for work that was not demonstrated, and never turn our preferred repository layout into a universal requirement.

Use it alongside How AI Agents Discover and Load Skills, Portable Agent Skills Across Copilot, Codex, and Claude, and How to Write Agent Skills That Work Reliably.

Want a repository audit with explicit evidence and compatibility boundaries? Explore ByblosAI or contact us.