Audit CLI help output against README documentation

[gregpriday]

Summary

Verify that the live copytree --help output exactly matches the flags, examples, and descriptions documented in the README to ensure consistency between documentation and implementation.

Problem Statement

As the CLI evolves, discrepancies can emerge between:

• Documented flags in README vs actual implementation
• Flag descriptions and examples
• Command aliases and shortcuts
• Default values and behavior

This affects:

• New users learning the tool from documentation
• Documentation accuracy and trust
• Support burden from incorrect examples
• Developer experience

Current State

Documentation Sources:

• README.md - User-facing documentation
  • Frequently Used Flags section
  • Common Recipes section
  • Quick Start examples
• bin/copytree.js - CLI implementation with Commander.js
• docs/cli/copytree-reference.md - Detailed CLI reference

Known Areas to Verify:

• All 8 commands exist and work: copy, profile:list, profile:validate, copy:docs, config:validate, config:inspect, cache:clear, install:copytree
• Flag aliases match (-i/--display, -S/--stream, -t/--only-tree, -r/--as-reference)
• Default formats (XML is default)
• Format options (xml, json, markdown, tree)
• Git integration flags (-m/--modified, -c/--changed)
• Output destinations (clipboard, file, display, stream)
• All documented examples are runnable

Proposed Solution

Create systematic audit to verify:

1. All documented flags exist in CLI implementation
2. All CLI flags are documented
3. Descriptions match between docs and --help
4. Examples in README work as written
5. Aliases are consistent
6. Default values are correct

Affected Components

• bin/copytree.js - CLI command definitions
• README.md - Main documentation
• docs/cli/copytree-reference.md - CLI reference docs

Implementation Approach

1. Create Automated Test:

   • Create tests/e2e/cli-help.test.js
   • Snapshot test for copytree --help output
   • Test for each command's help
   • Verify flag presence

2. Manual Audit Checklist:

   • Compare README "Frequently Used Flags" vs actual options
   • Test each example from "Common Recipes"
   • Verify all format options work
   • Check flag aliases
   • Verify default behaviors

3. Update as Needed:

   • Fix discrepancies (prefer implementation over docs if both work)
   • Add missing documentation
   • Remove outdated flags from docs
   • Update examples to match current behavior

Tasks

• Create tests/e2e/cli-help.test.js with snapshot tests for help output
• Test copytree --help matches README introduction
• Test copytree copy --help matches documented copy command flags
• Test all 8 commands have help: profile:list, profile:validate, copy:docs, config:validate, config:inspect, cache:clear, install:copytree
• Verify all flags in "Frequently Used Flags" exist and work
• Test all examples from "Common Recipes" section
• Verify format options: --format xml, --format json, --format markdown, --format tree
• Verify git flags: -m/--modified, -c/--changed <ref>
• Verify output flags: -i/--display, -o/--output, -S/--stream, --clipboard
• Verify special flags: -t/--only-tree, -r/--as-reference, --with-line-numbers, --with-git-status
• Check that undocumented flags are intentional (internal/experimental)
• Update README if implementation is correct but docs are wrong
• Update CLI descriptions if docs are correct but implementation is unclear
• Add missing aliases if commonly needed
• Document any breaking changes from mismatches

Acceptance Criteria

• All documented flags exist in copytree --help output
• All examples in README can be copy-pasted and work
• CLI help text matches README descriptions
• No undocumented flags appear in help (unless intentionally internal)
• Snapshot tests prevent future drift between docs and implementation
• Default values documented correctly (format: xml, output: clipboard)
• All flag aliases work as documented
• All 8 commands appear in help and work

Additional Context

Flags documented in README to verify:

• --format <xml|json|markdown|tree> - Output format (default: xml)
• -t, --only-tree - Tree structure only
• -i, --display - Print to terminal
• --clipboard - Force clipboard
• -S, --stream - Stream output
• -C, --char-limit <n> - Character budget
• --with-line-numbers - Line numbers
• --info - File metadata
• --show-size - File sizes
• --with-git-status - Git status
• -r, --as-reference - Generate reference
• --always <pattern...> - Force include
• --dedupe - Remove duplicates
• --sort <path|size|modified|name|extension> - Sort files

Commands to verify:

1. copytree [path] - Main copy command (default)
2. copytree profile:list - List profiles
3. copytree profile:validate <name> - Validate profile
4. copytree cache:clear - Clear caches
5. copytree config:validate - Validate config
6. copytree config:inspect - Inspect config
7. copytree copy:docs - Copy documentation
8. copytree install:copytree - Installation wizard

Good Examples of CLI/Docs Parity:

• Prettier's CLI docs - Comprehensive flag documentation
• ESLint's CLI reference - Detailed examples

[gregpriday]

Goal

Pass if CLI help text and README/reference content are provably in lockstep; fail with precise diffs when any flag/alias/default/example diverges. Explicitly handle that the install:copytree command has been removed and must not appear in docs/tests.

Approach Overview

1. Stabilize Commander help output in bin/copytree.js and expose a machine-readable inventory of commands/options.
2. Add a tiny HelpIntrospector utility (reads Commander program/commands/options) to avoid parsing stdout.
3. Fence authoritative doc blocks in README.md and docs/cli/copytree-reference.md for exact comparison.
4. Add E2E tests to snapshot normalized --help for root and each subcommand; assert structural parity (existence, aliases, defaults, choices).
5. Add a sandbox runner to execute README examples (no side effects).
6. Update docs and tests to reflect that install:copytree is removed.
7. Wire parity checks into CI with clear failure artifacts.

Step-by-Step Plan

1. Stabilize help rendering

   • In bin/copytree.js set deterministic help: program.configureHelp({ sortOptions: true, sortSubcommands: true }); ensure every Option sets explicit default and choices (e.g., --format <format> with default xml and choices xml|markdown|json|ndjson|sarif|tree as actually supported).
   • Expected outcome: copytree --help and copytree <cmd> --help are stable across envs.

2. Create HelpIntrospector

   • Add src/cli/help-introspector.js with:

     • getCommands() → { name, aliases }[]
     • getOptions(cmd) → { short, long, arg, description, default, choices, hidden }[]
     • getHelp(cmd) → exact Commander help string

   • Wire it by importing the same Commander program instance you export from bin/copytree.js (export a factory or a builder).

   • Expected outcome: tests can assert flags/aliases/defaults without string scraping.

3. Fence docs for exact compare

   • Wrap the “Frequently Used Flags”, “Quick Start”, and “Common Recipes” sections with HTML comments:

     • <!-- copytree:flags:start --> … <!-- copytree:flags:end -->
     • <!-- copytree:quickstart:start --> … <!-- copytree:quickstart:end -->
     • <!-- copytree:recipes:start --> … <!-- copytree:recipes:end -->

   • Do the same for docs/cli/copytree-reference.md where detailed per-command help is documented.

   • Expected outcome: tests can extract the exact, authoritative blocks to compare.

4. E2E parity tests

   • Add tests/e2e/cli-help.test.js and tests/helpers/cli.js to exec node bin/copytree.js with FORCE_COLOR=0, NO_COLOR=1, COLUMNS=100, LANG=C.

   • Snapshot normalized help for:

     • root, copy, profile:list, profile:validate, copy:docs, config:validate, config:inspect, cache:clear

   • Structural assertions via HelpIntrospector:

     • Flag presence and aliases: -i/--display, -S/--stream, -t/--only-tree, -r/--as-reference, -m/--modified, -c/--changed <ref>, -o/--output, --clipboard, -C/--char-limit <n>, --with-line-numbers, --info, --show-size, --with-git-status, --always <pattern...>, --dedupe, --sort <path|size|modified|name|extension>.
     • Default/choices parity: --format accepts xml|markdown|json|ndjson|sarif|tree with default xml.

   • Expected outcome: red/green parity with clear diffs.

5. README recipes runner

   • Add tests/e2e/recipes.test.js that:

     • Materializes a temp repo fixture to exercise -m/--modified and -c/--changed.
     • Forces terminal output (--display) or temp files (-o) to avoid clipboard dependence.
     • Executes every fenced command line and asserts zero exit plus non-empty output or expected file creation.

   • Expected outcome: all README examples are copy-paste runnable.

6. Remove install:copytree from docs/tests

   • Update README.md and docs/cli/copytree-reference.md: list exactly these commands: copy, profile:list, profile:validate, copy:docs, config:validate, config:inspect, cache:clear.
   • Update acceptance and tasks in the issue to 7 commands.
   • Expected outcome: no stale references to the removed command.

7. CI wiring

   • Add a docs-parity job in .github/workflows/ci.yml running the new E2E parity suite and exporting artifacts on failure (actual vs expected help, extracted doc blocks).
   • Expected outcome: drift is blocked at PR time.

Tests

• Unit

  • src/cli/help-introspector.test.js:

    • returns 7 commands (no install:copytree)
    • copy exposes documented options and defaults (format default xml; accepts xml|markdown|json|ndjson|sarif|tree)

• E2E: help parity

  • tests/e2e/cli-help.test.js:

    • snapshots: root and each subcommand
    • structural parity vs fenced docs for flags, aliases, defaults, choices
    • fails if any public option appears in help but not in fenced docs (allow internal hidden options if marked hidden)

• E2E: recipes

  • tests/e2e/recipes.test.js:

    • executes each README recipe in a temp repo; asserts stdout/files as appropriate

Edge Cases & Risks

• Commander wrapping differences (COLUMNS): normalize whitespace in snapshots.
• Hidden/internal flags: mark them hidden in Commander or add an explicit allowlist in tests.
• Locale/time variance: set LANG=C, TZ=UTC during tests.
• Clipboard availability: always force --display/-o in tests.

Rollout & Monitoring

• Feature flag: environment variable COPYTREE_DOCS_PARITY=strict|relaxed|off consumed by tests to optionally skip parity locally.
• Telemetry (log-only): on parity failure, write a machine-readable diff under docs/telemetry/.
• Rollback: if parity job becomes too noisy, set COPYTREE_DOCS_PARITY=off (single toggle) or revert the last doc/help change.
• Abort criteria: block release if ≥3 mismatches for any single command or any quick-start example fails twice on develop.

Acceptance Checklist

• Parity: root and each subcommand --help match fenced blocks; no install:copytree anywhere.
• Flags/aliases/defaults/choices: exact match (including --format default xml; accepts xml/markdown/json/ndjson/sarif/tree).
• Examples: every README recipe runs in a sandbox and produces output.
• Tests: green locally and in CI; parity artifacts uploaded on failures.
• SLO/Alert: p95 copytree --help ≤ 150 ms in CI; alert when any parity mismatch is pushed to develop.
• Logs/Metrics: parity mismatch reports under docs/telemetry/.

Naming note: keep module and metric names consistent and lower-snake or kebab where used:

• Utility: help-introspector
• Tests: cli-help.test.js, recipes.test.js
• Metrics/events: cli.docs_parity.failure, cli.docs_parity.mismatch_count, cli.recipes.ok

Files referenced:

• bin/copytree.js
• README.md
• docs/cli/copytree-reference.md
• tests/e2e/cli-help.test.js
• tests/e2e/recipes.test.js
• tests/helpers/cli.js
• src/cli/help-introspector.js