fix(scripts): align agent frontmatter schema with VS Code spec #469
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #469 +/- ##
==========================================
+ Coverage 83.41% 83.75% +0.33%
==========================================
Files 20 20
Lines 3510 3583 +73
==========================================
+ Hits 2928 3001 +73
Misses 582 582
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Updates the repo’s agent frontmatter JSON schema to match VS Code Custom Agents spec deltas from #425, and extends the PowerShell schema validator to enforce the newly needed schema constructs during frontmatter linting.
Changes:
- Expanded
agent-frontmatter.schema.jsonwithagents,user-invokable,disable-model-invocation,handoffs.model, and correctedmodelto allow string or string-array. - Enhanced
Test-JsonSchemaValidationto supportoneOfand nestedobject/array.itemsvalidation patterns used by repo schemas. - Added unit tests for
oneOfand nested object/array validation scenarios.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/tests/linting/Validate-MarkdownFrontmatter.Tests.ps1 | Adds unit tests covering oneOf and nested object/array schema validation behavior. |
| scripts/linting/schemas/agent-frontmatter.schema.json | Aligns agent frontmatter schema fields and types with the VS Code spec deltas (while preserving maturity). |
| scripts/linting/Validate-MarkdownFrontmatter.ps1 | Implements validator support for oneOf and nested object/array validation patterns used by the updated schema. |
| .gitignore | Ignores locally generated dependency-pinning artifacts directory. |
| .cspell.json | Adds “invokable” to spelling allowlist. |
- Address Copilot review: update validation coverage docs and avoid array += in loops - Fix array conversion to preserve single-item arrays - Add tests to cover oneOf + nested object/array conversion paths (Codecov patch coverage)
| try { | ||
| if ($SchemaContent.required) { | ||
| foreach ($requiredField in $SchemaContent.required) { | ||
| if (-not $Frontmatter.ContainsKey($requiredField)) { | ||
| $errors.Add("Missing required field: $requiredField") | ||
| function ConvertTo-HashTable { | ||
| <# | ||
| .SYNOPSIS | ||
| Converts a PSCustomObject or hashtable to a hashtable recursively. | ||
| #> | ||
| [CmdletBinding()] | ||
| [OutputType([hashtable])] | ||
| param( | ||
| [Parameter(Mandatory = $true)] | ||
| [ValidateScript({ $_ -is [hashtable] -or $_ -is [pscustomobject] })] | ||
| [object]$InputObject | ||
| ) | ||
|
|
||
| function ConvertTo-ObjectArray { | ||
| <# | ||
| .SYNOPSIS | ||
| Converts an enumerable to an object array, converting nested objects to hashtables. | ||
| #> | ||
| [CmdletBinding()] | ||
| [OutputType([object[]])] | ||
| param( | ||
| [Parameter(Mandatory = $true)] | ||
| [System.Collections.IEnumerable]$Enumerable | ||
| ) | ||
|
|
||
| $list = [System.Collections.Generic.List[object]]::new() | ||
| foreach ($item in $Enumerable) { | ||
| if ($item -is [pscustomobject] -or $item -is [hashtable]) { | ||
| $list.Add((ConvertTo-HashTable -InputObject $item)) | ||
| } | ||
| else { | ||
| $list.Add($item) | ||
| } | ||
| } | ||
|
|
||
| # Prevent PowerShell from unrolling single-element arrays when used in expressions/assignments. | ||
| return ,$list.ToArray() | ||
| } |
There was a problem hiding this comment.
Test-JsonSchemaValidation defines helper functions (ConvertTo-HashTable, Test-ValueAgainstSchema, and the nested ConvertTo-ObjectArray) inside the function body. Since this validator is called per-file, repeatedly redefining these functions can add noticeable overhead when validating many markdown files. Consider moving these helpers to script scope / module scope (or caching them) so they’re defined once per script load rather than once per invocation.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Align disable-model-invocation description with microsoft#425/VS Code intent - Aggregate oneOf mismatch errors across subschemas for stable, actionable diagnostics
Exclude IDictionary/hashtable from array type detection to avoid false passes and confusing item errors.
katriendg
left a comment
katriendg
left a comment
There was a problem hiding this comment.
Thank you so much for picking up an open issue and addressing this @aavetis!
Really appreciate your work on this. Love the code coverage result as well, neat!
I have a few minor comments, if you could please review those? For the rest I have no other comments and we look forward to being able to merge this one in.
Thanks again and hope to see you contribute in the future.
| return $localErrors.ToArray() | ||
| } | ||
|
|
||
| # Enum validation. |
There was a problem hiding this comment.
Could we move the Enum validation to after the type checks? Type checking before will provide more accurate information and if type is correct then the enum check is valuable.
So just move the block and add a final return $localErrors.ToArray() for the function to return.
| $result = Test-JsonSchemaValidation -Frontmatter $frontmatter -SchemaContent $script:NestedSchema | ||
| $result.IsValid | Should -BeTrue | ||
| } | ||
|
|
There was a problem hiding this comment.
nit: could you add one Rejects any non "*" string values explicitly?
| foreach ($sub in $Schema.oneOf) { | ||
| $subErrs = Test-ValueAgainstSchema -Value $Value -Schema $sub -Path $Path | ||
| if ($subErrs.Count -eq 0) { | ||
| $passCount++ |
There was a problem hiding this comment.
nit: consider breaking once the result hits 2 and becomes invalid, so we don't need to further evaluate:
| $passCount++ | |
| $passCount++ | |
| if ($passCount -gt 1) { break } |
|
@aavetis - once the code comments are resolved, I'll get this approved and merged in. Thanks for the contribution!!! |
| - oneOf: Supported (exactly one subschema must match) | ||
| - object: Supported for required/properties/items patterns used in repo schemas |
There was a problem hiding this comment.
The docblock’s “Limitations” section lists oneOf as a limitation while also stating it’s supported. Consider moving oneOf out of “Limitations” (or rewording the section) to avoid contradictory documentation for the validator’s capabilities.
| - oneOf: Supported (exactly one subschema must match) | |
| - object: Supported for required/properties/items patterns used in repo schemas |
4 participants
Description
Aligns the agent frontmatter JSON schema with the VS Code Custom Agents spec deltas described in #425, and updates the repo's schema validator so the new schema constraints are enforced.
Issue #425 Requirements (Checklist)
agent-frontmatter.schema.json:agents,user-invokable,disable-model-invocationmodeltype to allow string OR array of stringshandoffs.promptoptionalhandoffs.model(optional string)maturityfieldImplementation Notes
Test-JsonSchemaValidationnow supports the schema constructs required for this issue (oneOf, nestedobjectwithrequired/properties, andarray.items).additionalPropertiesis still not enforced by the validator (documented), consistent with prior behavior.Testing
Ran locally:
npm run lint:frontmatternpm run lint:psnpm run test:psnpm run spell-checkpwsh -NoProfile -Command "& scripts/linting/Validate-MarkdownFrontmatter.ps1 -WarningsAsErrors -EnableSchemaValidation -Files (Get-ChildItem -Path .github/agents -Filter *.agent.md -File | Select-Object -ExpandProperty FullName)"Fixes #425
Notes
dependency-pinning-artifacts/to.gitignoresince it is generated by the dependency pinning scan script during local runs and can otherwise leave the working tree dirty.