diff --git a/_bmad-output/test-design-architecture.md b/_bmad-output/test-design-architecture.md new file mode 100644 index 0000000..12450e5 --- /dev/null +++ b/_bmad-output/test-design-architecture.md @@ -0,0 +1,260 @@ +# Test Design for Architecture: QA Tester Process Optimization (TRA) + +**Purpose:** Architectural concerns, testability gaps, and NFR requirements for review by Architecture/Dev teams. Serves as a contract between QA and Engineering on what must be addressed before test development begins. + +**Date:** 2026-02-06 +**Author:** Edouard Zemb (via TEA Test Architect) +**Status:** Architecture Review Pending +**Project:** test-framework +**PRD Reference:** `_bmad-output/planning-artifacts/prd.md` +**ADR Reference:** `_bmad-output/planning-artifacts/architecture.md` + +--- + +## Executive Summary + +**Scope:** CLI tool (Rust) for QA process automation — ingestion Jira/Squash, checklist testability, scoring Go/Clarify/No-Go, test design assistance (LLM), anomaly management, reporting (CR/PPT/exports), with mandatory anonymization and audit logging. + +**Business Context** (from PRD): + +- **Impact:** Gain 2-3h/semaine, reduction erreurs reporting -50%, feedback client >= 4/5 +- **Problem:** Workflow QA TRA manuel, repetitif, fragile, sans standardisation +- **Target:** MVP operationnel multi-projets + +**Architecture** (from ADR): + +- **Key Decision 1:** Rust workspace multi-crates (10 crates: tf-cli, tf-core, tf-connectors, tf-storage, tf-security, tf-config, tf-logging, tf-diagnostics, tf-llm, tf-export) +- **Key Decision 2:** SQLite local chiffre + keyring OS pour secrets +- **Key Decision 3:** Anonymisation inline obligatoire avant tout envoi cloud LLM + +**Expected Scale:** Single-user CLI, volume standard: ~50-200 tickets/lot, ~10-20 cas de test/ticket, pipelines < 5 min. + +**Risk Summary:** + +- **Total risks**: 12 +- **High-priority (>=6)**: 4 risks requiring immediate mitigation +- **Test effort**: ~46 tests (~3-4 semaines pour 1 QA) + +--- + +## Quick Guide + +### BLOCKERS - Team Must Decide (Can't Proceed Without) + +**Sprint 0 Critical Path** - These MUST be completed before QA can write integration tests: + +1. **C2: Trait abstractions for connectors** - Define `trait Connector` in tf-connectors with mock implementations for Jira/Squash/SharePoint/Ollama (recommended owner: Dev) +2. **C1: Test data seeding module** - Implement `tf-storage/test_helpers.rs` with deterministic SQLite seeding functions (recommended owner: Dev) +3. **C5: Anonymization verification** - Implement PII canary test infrastructure in tf-security (recommended owner: Dev + QA) + +**What we need from team:** Complete these 3 items in Sprint 0 or test development is blocked. + +--- + +### HIGH PRIORITY - Team Should Validate (We Provide Recommendation, You Approve) + +1. **R-01: PII leak prevention** - Add regex-based PII scanner in tf-security/redact.rs + CI scan step (Sprint 0) +2. **R-06: Proxy/IT block fallback** - Validate CSV fallback is functionally equivalent end-to-end, not just a data format (Sprint 0-1) +3. **C3: Fault injection mechanism** - Add `--simulate-failure` flag for resilience testing of degraded mode (Sprint 1) + +**What we need from team:** Review recommendations and approve (or suggest changes). + +--- + +### INFO ONLY - Solutions Provided (Review, No Decisions Needed) + +1. **Test strategy**: 70% Unit (cargo test + mockall), 25% Integration (assert_cmd + test SQLite), 5% E2E CLI (assert_cmd full pipeline) +2. **Tooling**: cargo test, mockall 0.13, assert_cmd 2.0, predicates 3.1 +3. **Tiered CI/CD**: PR (<10 min all unit+integration), Nightly (E2E + golden files), Weekly (benchmarks + chaos) +4. **Coverage**: ~46 test scenarios prioritized P0-P3 with risk-based classification +5. **Quality gates**: P0=100%, P1>=95%, coverage >=80% on core crates + +**What we need from team:** Just review and acknowledge. + +--- + +## For Architects and Devs - Open Topics + +### Risk Assessment + +**Total risks identified**: 12 (4 high-priority score >=6, 6 medium, 2 low) + +#### High-Priority Risks (Score >=6) - IMMEDIATE ATTENTION + +| Risk ID | Category | Description | Prob | Impact | Score | Mitigation | Owner | Timeline | +|---------|----------|-------------|------|--------|-------|------------|-------|----------| +| **R-01** | **SEC** | Fuite PII vers LLM cloud (anonymisation incomplete) | 2 | 3 | **6** | Test canary PII + revue pipeline anonymize.rs | tf-security | Sprint 0 | +| **R-02** | **SEC** | Secrets en clair dans logs/config/repo | 2 | 3 | **6** | Scan CI automatique + tests unitaires redact.rs | tf-security | Sprint 0 | +| **R-03** | **TECH** | Connecteurs non mockables - tests lents/flaky | 3 | 2 | **6** | Trait abstrait + impl mock par connecteur | tf-connectors | Sprint 0 | +| **R-06** | **OPS** | Proxy IT bloque appels API Jira/Squash | 2 | 3 | **6** | Mode degrade CSV automatique + diagnostic explicite | tf-connectors | Sprint 0-1 | + +#### Medium-Priority Risks (Score 3-5) + +| Risk ID | Category | Description | Prob | Impact | Score | Mitigation | Owner | +|---------|----------|-------------|------|--------|-------|------------|-------| +| R-04 | DATA | Corruption SQLite mid-pipeline | 2 | 2 | 4 | Transactions + test recovery | tf-storage | +| R-07 | BUS | Templates non alignes client | 2 | 2 | 4 | Golden file tests | tf-export | +| R-08 | TECH | Generation OOXML fragile | 2 | 2 | 4 | Golden file + validation ZIP/XML | tf-export | +| R-09 | SEC | Keyring indisponible en CI | 2 | 2 | 4 | Mock keyring + fallback env var | tf-security | +| R-10 | DATA | Mapping Jira/Squash divergents | 2 | 2 | 4 | Validation stricte au load | tf-connectors | +| R-11 | OPS | Purge locale ne s'execute pas | 2 | 2 | 4 | Test purge integre + alerte | tf-storage | + +#### Low-Priority Risks (Score 1-2) + +| Risk ID | Category | Description | Prob | Impact | Score | Action | +|---------|----------|-------------|------|--------|-------|--------| +| R-05 | PERF | CLI depasse 2s (cold start Rust) | 1 | 2 | 2 | Monitor via benchmark CI | +| R-12 | TECH | Ollama indisponible | 2 | 1 | 2 | Message explicite + hint | + +--- + +### Testability Concerns and Architectural Gaps + +**ACTIONABLE CONCERNS - Architecture Team Must Address** + +#### 1. Blockers to Fast Feedback + +| Concern | Impact | What Architecture Must Provide | Owner | Timeline | +|---------|--------|-------------------------------|-------|----------| +| **No trait abstraction for connectors** | Cannot mock Jira/Squash/SharePoint/Ollama - tests require live services | Define `trait ConnectorClient` in tf-connectors + MockJiraClient, MockSquashClient impls | Dev | Sprint 0 | +| **No test data seeding** | Cannot inject specific SQLite states (tickets importes, config edge cases) | `tf-storage/test_helpers.rs` with `seed_tickets()`, `seed_config()` functions | Dev | Sprint 0 | +| **No PII verification mechanism** | Cannot automatically verify anonymization completeness | PII pattern scanner in tf-security callable from tests | Dev | Sprint 0 | + +#### 2. Architectural Improvements Needed + +1. **Fault injection for degraded mode** + - **Current problem**: No way to simulate API failures in tests + - **Required change**: Add `--simulate-failure jira|squash|ollama` flag or config override + - **Impact if not fixed**: Degraded mode (NFR9) untestable without real outages + - **Owner**: Dev + - **Timeline**: Sprint 1 + +2. **Performance observability** + - **Current problem**: No measurable pipeline duration metrics + - **Required change**: Add `tracing` spans with duration for each pipeline stage + - **Impact if not fixed**: NFR6 (CR <2min) and NFR8 (CLI <2s) not verifiable in CI + - **Owner**: Dev + - **Timeline**: Sprint 1 + +3. **Purge TTL specification** + - **Current problem**: "TTL court" mentioned but no concrete value + - **Required change**: Set TTL = 24h in config schema, implement purge function + - **Impact if not fixed**: NFR5 (purge <24h) not testable + - **Owner**: Dev + - **Timeline**: Sprint 0 + +--- + +### Testability Assessment Summary + +#### What Works Well + +- Workspace multi-crates: each crate testable independently with clear boundaries +- CLI 100% headless: all business logic accessible sans UI, tests unitaires rapides +- Erreurs structurees (code + message + hint): assertions sur cas d'erreur simples +- Codes de sortie normalises (0/1/2/3): validation CLI deterministe +- Config externalisee (YAML + profils): permutation de configs de test triviale +- Rust type system: modeles types + validation stricte - bugs structurels elimines au compile-time +- Fallback CSV prevu architecturalement: tests offline possibles + +#### Accepted Trade-offs (No Action Required) + +- **Pas de metriques Prometheus/Datadog**: CLI single-user, logs JSON suffisants pour le MVP +- **Pas de tracing distribue (W3C)**: architecture monolithique locale, non necessaire + +--- + +### Risk Mitigation Plans (High-Priority Risks >=6) + +#### R-01: Fuite PII vers LLM cloud (Score: 6) - HIGH + +**Mitigation Strategy:** + +1. Implementer regex-based PII scanner dans `tf-security/anonymize.rs` (emails, noms, tokens, numeros) +2. Ajouter un hook pre-envoi dans `tf-llm/orchestrator.rs` qui bloque si PII detectee post-anonymisation +3. Integrer un scan CI qui verifie 0 pattern PII dans les sorties de test cloud + +**Owner:** tf-security (Dev) +**Timeline:** Sprint 0 +**Status:** Planned +**Verification:** Test canary injectant des PII connues et verifiant leur masquage en sortie + +#### R-02: Secrets en clair (Score: 6) - HIGH + +**Mitigation Strategy:** + +1. Implementer `redact.rs` avec filtrage systematique des patterns secrets dans les logs +2. Ajouter un scan CI grep sur les fichiers de config et le repo (0 match autorise) +3. Tests unitaires verifiant que keyring.rs ne retourne jamais de secret dans les messages d'erreur + +**Owner:** tf-security (Dev) +**Timeline:** Sprint 0 +**Status:** Planned +**Verification:** Tests unitaires + scan CI automatique + +#### R-03: Connecteurs non mockables (Score: 6) - HIGH + +**Mitigation Strategy:** + +1. Definir `trait ConnectorClient` avec methodes async pour chaque operation (import, read, write) +2. Implementer `MockJiraClient`, `MockSquashClient`, `MockSharePointClient` via mockall +3. Injecter les mocks via dependency injection dans tf-core/pipeline + +**Owner:** tf-connectors (Dev) +**Timeline:** Sprint 0 +**Status:** Planned +**Verification:** Tests unitaires du pipeline avec 100% mocks, 0 appel reseau + +#### R-06: Proxy IT bloque APIs (Score: 6) - HIGH + +**Mitigation Strategy:** + +1. Implementer detection automatique d'echec de connexion (timeout + HTTP errors) +2. Basculement automatique vers CSV avec message explicite (cause + action) +3. Valider que le mode CSV produit des resultats fonctionnellement equivalents + +**Owner:** tf-connectors (Dev) +**Timeline:** Sprint 0-1 +**Status:** Planned +**Verification:** Test E2E avec mock serveur down -> fallback CSV -> meme sortie + +--- + +### Assumptions and Dependencies + +#### Assumptions + +1. Le workspace Rust et la structure multi-crates sont en place avant le Sprint 0 test +2. Les crates de base (tf-config, tf-logging) sont fonctionnelles et testees unitairement +3. Les APIs Jira et Squash restent stables (pas de breaking changes) pendant le MVP +4. Le keyring OS est disponible sur le poste de dev; en CI, un mock est acceptable + +#### Dependencies + +1. **Trait abstractions (R-03)** - Required by Sprint 0 +2. **Test data seeding module (C1)** - Required by Sprint 0 +3. **PII scanner (R-01)** - Required by Sprint 0 +4. **Purge TTL implementation (C6)** - Required by Sprint 0 + +#### Risks to Plan + +- **Risk**: Sprint 0 blockers non termines a temps + - **Impact**: QA ne peut pas ecrire de tests d'integration + - **Contingency**: Commencer par les tests unitaires de tf-config et tf-security (0 dependance externe) + +--- + +**End of Architecture Document** + +**Next Steps for Architecture Team:** + +1. Review Quick Guide and prioritize Sprint 0 blockers (trait abstractions, seeding, PII scanner) +2. Assign owners and timelines for high-priority risks R-01, R-02, R-03, R-06 +3. Validate assumptions (keyring mock en CI, APIs stables) +4. Provide feedback to QA on testability gaps + +**Next Steps for QA Team:** + +1. Wait for Sprint 0 blockers to be resolved +2. Refer to companion QA doc (test-design-qa.md) for test scenarios +3. Begin test infrastructure setup (test helpers, mock implementations) diff --git a/_bmad-output/test-design-qa.md b/_bmad-output/test-design-qa.md new file mode 100644 index 0000000..24adc35 --- /dev/null +++ b/_bmad-output/test-design-qa.md @@ -0,0 +1,377 @@ +# Test Design for QA: QA Tester Process Optimization (TRA) + +**Purpose:** Test execution recipe for QA team. Defines what to test, how to test it, and what QA needs from other teams. + +**Date:** 2026-02-06 +**Author:** Edouard Zemb (via TEA Test Architect) +**Status:** Draft +**Project:** test-framework + +**Related:** See Architecture doc (test-design-architecture.md) for testability concerns and architectural blockers. + +--- + +## Executive Summary + +**Scope:** Complete test coverage for test-framework Rust CLI tool: 10 crates, 31 FRs, 16 NFRs, 8 epics across config, security, connectors, storage, LLM, export, core pipeline, and CLI. + +**Risk Summary:** + +- Total Risks: 12 (4 high-priority score >=6, 6 medium, 2 low) +- Critical Categories: SEC (anonymization, secrets), TECH (mockability), OPS (proxy fallback) + +**Coverage Summary:** + +- P0 tests: ~12 (security, anonymization, fallback CSV, LLM local-only) +- P1 tests: ~28 (config, logging, storage, connectors, LLM, export, core pipeline, CLI) +- P2 tests: ~5 (cache, grouping, template guards, benchmarks, diagnostics) +- P3 tests: ~1 (CLI cold start benchmark) +- **Total**: ~46 tests (~3-4 weeks with 1 QA) + +--- + +## Dependencies & Test Blockers + +**CRITICAL:** QA cannot proceed without these items from other teams. + +### Backend/Architecture Dependencies (Sprint 0) + +**Source:** See Architecture doc "Quick Guide" for detailed mitigation plans + +1. **Trait abstractions for connectors (R-03)** - Dev - Sprint 0 + - QA needs `MockJiraClient`, `MockSquashClient`, `MockOllamaClient` via mockall + - Blocks all connector and pipeline integration tests + +2. **Test data seeding module (C1)** - Dev - Sprint 0 + - QA needs `tf-storage/test_helpers.rs` with `seed_tickets()`, `seed_config()` functions + - Blocks integration tests requiring specific SQLite states + +3. **PII verification scanner (R-01)** - Dev - Sprint 0 + - QA needs callable PII pattern scanner in tf-security + - Blocks canary tests for anonymization validation + +### QA Infrastructure Setup (Sprint 0) + +1. **Test Helper Module** - QA + - `tests/common/mod.rs` with shared test fixtures + - Temporary SQLite database per test (parallel-safe) + - Config builder for test scenarios + +2. **Test Environments** - QA + - Local: `cargo test` with in-memory/temp SQLite + - CI/CD: GitHub Actions with `cargo test` + `cargo clippy` + - No staging needed (single-user CLI) + +**Example test pattern (Rust):** + +```rust +// tests/common/mod.rs +use tf_config::Config; +use tf_storage::test_helpers::TestDb; + +pub fn test_config() -> Config { + Config::from_str(r#" + project_name: "test-project" + profile: "default" + output_folder: "/tmp/test-output" + "#).unwrap() +} + +pub fn test_db() -> TestDb { + TestDb::new_temp() // Auto-cleanup on drop +} + +// tests/unit/config_test.rs +#[test] +fn valid_config_loads_all_fields() { + let config = Config::from_file("tests/fixtures/valid_config.yaml").unwrap(); + assert_eq!(config.project_name, "test-project"); + assert!(config.output_folder.exists()); +} + +#[test] +fn invalid_config_returns_structured_error() { + let err = Config::from_file("tests/fixtures/missing_field.yaml").unwrap_err(); + assert_eq!(err.code(), "VALIDATION_ERROR"); + assert!(err.message().contains("project_name")); + assert!(err.hint().contains("add project_name")); +} +``` + +--- + +## Risk Assessment + +**Note:** Full risk details in Architecture doc. This section summarizes risks relevant to QA test planning. + +### High-Priority Risks (Score >=6) + +| Risk ID | Category | Description | Score | QA Test Coverage | +|---------|----------|-------------|-------|-----------------| +| **R-01** | SEC | Fuite PII vers cloud | **6** | SEC-INT-001: canary test injectant PII connues, verifiant masquage | +| **R-02** | SEC | Secrets en clair | **6** | SEC-INT-002: scan repo/config + tests unitaires redact | +| **R-03** | TECH | Connecteurs non mockables | **6** | CON-UNIT-*: tous les tests connectors via mocks | +| **R-06** | OPS | Proxy bloque APIs | **6** | CON-INT-001: test fallback CSV end-to-end | + +### Medium/Low-Priority Risks + +| Risk ID | Category | Description | Score | QA Test Coverage | +|---------|----------|-------------|-------|-----------------| +| R-04 | DATA | Corruption SQLite | 4 | STO-INT-001: crash mid-write recovery | +| R-07 | BUS | Templates non alignes | 4 | EXP-INT-001: golden file Excel | +| R-08 | TECH | OOXML fragile | 4 | EXP-INT-002: golden file PPT | +| R-09 | SEC | Keyring indisponible CI | 4 | SEC-UNIT-006: mock keyring | +| R-10 | DATA | Mapping divergents | 4 | CON-UNIT-006: validation stricte | +| R-11 | OPS | Purge ne s'execute pas | 4 | STO-UNIT-003: test TTL/purge | +| R-05 | PERF | CLI >2s | 2 | CLI-E2E-005: benchmark | +| R-12 | TECH | Ollama indisponible | 2 | LLM-UNIT-004: erreur explicite | + +--- + +## Test Coverage Plan + +**IMPORTANT:** P0/P1/P2/P3 = **priority and risk level** (what to focus on if time-constrained), NOT execution timing. See "Execution Strategy" for when tests run. + +### P0 (Critical) + +**Criteria:** Blocks core functionality + High risk (>=6) + No workaround + Security/compliance critical + +| Test ID | Requirement | Test Level | Risk Link | Notes | +|---------|-------------|------------|-----------|-------| +| **SEC-UNIT-001** | NFR1: Anonymisation emails/noms/tokens | Unit | R-01 | Regex patterns PII | +| **SEC-UNIT-003** | NFR1: Anonymisation echoue -> envoi bloque | Unit | R-01 | Fail-safe | +| **SEC-UNIT-004** | NFR4: Logs ne contiennent aucun PII | Unit | R-02 | Redact patterns | +| **SEC-INT-001** | NFR1: Pipeline complet donnee -> anonymize -> cloud mock -> sortie masquee | Integration | R-01 | Test canary | +| **SEC-INT-002** | NFR2: Scan repo/config 0 secret en clair | Integration | R-02 | CI scan | +| **CON-UNIT-004** | NFR9: CSV fallback equivalent fonctionnel | Unit | R-06 | Mode degrade | +| **CON-INT-001** | NFR9: API Jira indisponible -> fallback CSV transparent | Integration | R-06 | End-to-end fallback | +| **LLM-UNIT-002** | NFR1: Mode cloud anonymisation appelee avant envoi | Unit | R-01 | Pre-envoi check | +| **LLM-UNIT-005** | FR29: Mode local uniquement 0 requete cloud | Unit | - | Epic 7 compliance | +| **LOG-UNIT-002** | NFR4: Champs sensibles masques dans logs | Unit | R-02 | ASR-4 | +| **CLI-E2E-004** | NFR9: API indisponible -> degrade CSV -> exit 0 + warning | E2E | R-06 | Journey complet | + +**Total P0:** ~11 tests + +--- + +### P1 (High) + +**Criteria:** Core user journeys + Medium risk + Frequent usage + +| Test ID | Requirement | Test Level | Risk Link | Notes | +|---------|-------------|------------|-----------|-------| +| **CFG-UNIT-001** | FR23: Config.yaml valide -> struct Config | Unit | - | Fondation | +| **CFG-UNIT-002** | FR23: Config invalide -> erreur structuree | Unit | - | UX erreur | +| **CFG-UNIT-003** | FR24: Selection profil existant | Unit | - | Multi-profil | +| **CFG-UNIT-004** | FR24: Profil inconnu -> erreur + liste | Unit | - | UX erreur | +| **CFG-UNIT-005** | FR25: Chargement template existant | Unit | - | Templates | +| **CFG-UNIT-006** | FR25: Template manquant -> erreur | Unit | - | Guard | +| **CFG-UNIT-007** | FR2/3: Checklist + regles scoring | Unit | - | Prerequis triage | +| **CFG-UNIT-008** | FR2/3: Checklist manquante -> erreur | Unit | - | Guard | +| **CFG-INT-001** | Config + profil -> pipeline init | Integration | - | Chaine init | +| **SEC-UNIT-002** | Pas de double anonymisation | Unit | - | Correctness | +| **SEC-UNIT-005** | FR31: Store/retrieve secret keyring | Unit | R-09 | Keyring | +| **SEC-UNIT-006** | Keyring indisponible -> erreur | Unit | R-09 | Fallback | +| **LOG-UNIT-001** | FR30: Log JSON structure genere | Unit | - | Baseline | +| **LOG-UNIT-003** | FR30: Logs dans dossier configure | Unit | - | Output path | +| **LOG-INT-001** | Execution CLI -> log valide JSON sans PII | Integration | - | Bout-en-bout | +| **STO-UNIT-001** | CRUD SQLite avec transactions | Unit | R-04 | Fondation | +| **STO-UNIT-002** | Chiffrement applicatif au repos | Unit | - | Security | +| **STO-UNIT-003** | NFR5: TTL/purge donnees >24h | Unit | R-11 | Purge | +| **STO-INT-001** | Crash mid-write -> recovery | Integration | R-04 | Resilience | +| **STO-INT-002** | NFR10: Replay derniere execution | Integration | - | Reprise | +| **CON-UNIT-001** | FR1: Import Jira -> struct Ticket[] | Unit (mock) | - | Core | +| **CON-UNIT-002** | Jira token invalide -> erreur + hint | Unit | - | UX | +| **CON-UNIT-003** | FR12/16: Squash read campagnes | Unit (mock) | - | Core | +| **CON-UNIT-005** | Retry exponentiel 3 tentatives | Unit | - | Pattern | +| **CON-UNIT-006** | Validation stricte mapping | Unit | R-10 | Data integrity | +| **CORE-UNIT-001** | FR2: Checklist testabilite | Unit | - | Triage | +| **CORE-UNIT-002** | FR3: Scoring Go/Clarify/No-Go | Unit | - | Triage | +| **CORE-UNIT-003** | FR4: Marquage clarification + cause | Unit | - | Tracabilite | +| **CORE-UNIT-005** | FR14/15: Brouillon anomalie | Unit | - | Template | +| **CORE-INT-001** | Pipeline triage: import -> checklist -> score | Integration | - | Journey 1 | +| **CORE-INT-002** | NFR6: Pipeline CR < 2 min | Integration | - | Performance | +| **EXP-UNIT-001** | FR18/22: Export Excel CR valide | Unit | R-07 | Reporting | +| **EXP-UNIT-002** | FR20/21: Export PPT OOXML | Unit | R-08 | Reporting | +| **EXP-UNIT-003** | FR22: Multi-formats json/csv/md/html/yaml | Unit | - | Exports | +| **EXP-INT-001** | Golden file Excel | Integration | R-07 | Validation | +| **EXP-INT-002** | Golden file PPT ZIP/XML | Integration | R-08 | Validation | +| **LLM-UNIT-001** | FR29: Mode local Ollama mock | Unit | - | LLM | +| **LLM-UNIT-003** | Mode auto: local OK -> local, local KO -> cloud | Unit | - | Orchestrator | +| **LLM-UNIT-004** | Ollama indisponible + cloud disabled -> erreur | Unit | R-12 | UX | +| **CLI-E2E-001** | tf triage -> sortie structuree exit 0 | E2E | - | Journey 1 | +| **CLI-E2E-002** | tf report -> fichier genere exit 0 | E2E | - | Journey 1 | +| **CLI-E2E-003** | Config invalide -> exit 2 + message | E2E | - | UX erreur | + +**Total P1:** ~31 tests + +--- + +### P2 (Medium) + +**Criteria:** Secondary features + Low risk + Edge cases + +| Test ID | Requirement | Test Level | Risk Link | Notes | +|---------|-------------|------------|-----------|-------| +| **STO-UNIT-004** | Cache hit/miss/expiration | Unit | - | Performance | +| **CORE-UNIT-004** | FR5: Regroupement tickets lot/perimetre | Unit | - | Secondary | +| **EXP-UNIT-004** | Template PPT manquant -> erreur | Unit | - | Guard | +| **CON-INT-002** | NFR7: Import perimetre < 5 min | Integration | - | Benchmark | +| **CLI-E2E-006** | tf diagnostics -> rapport sans PII | E2E | - | Ops | + +**Total P2:** ~5 tests + +--- + +### P3 (Low) + +**Criteria:** Nice-to-have + Benchmarks + +| Test ID | Requirement | Test Level | Notes | +|---------|-------------|------------|-------| +| **CLI-E2E-005** | NFR8: Commande simple < 2s | E2E (benchmark) | Cold start monitoring | + +**Total P3:** ~1 test + +--- + +## Execution Strategy + +**Philosophy:** Run everything in PRs unless there's significant infrastructure overhead. Cargo test with parallel execution is fast (100+ tests in <5 min for Rust unit tests). + +**Organized by TOOL TYPE:** + +### Every PR: cargo test (~5-10 min) + +**All functional tests** (from any priority level): + +- All unit tests (`cargo test --lib`) +- All integration tests (`cargo test --test '*'`) +- Clippy + format check (`cargo clippy && cargo fmt --check`) +- Parallelized natively by cargo (per-crate + per-test) +- Total: ~46 tests (includes P0, P1, P2) + +**Why run in PRs:** Rust tests are fast, no external infrastructure needed (all mocked) + +### Nightly: E2E + Golden Files (~15-30 min) + +**Full CLI E2E tests** (from any priority level): + +- CLI end-to-end via `assert_cmd` (tf triage, tf report, tf diagnostics) +- Golden file comparisons (Excel, PPT structure) +- Total: ~8 tests (CLI-E2E-*, EXP-INT-*) + +**Why defer to nightly:** File I/O intensive, longer setup, golden file comparisons + +### Weekly: Benchmarks + Chaos (~30-60 min) + +**Performance and resilience tests**: + +- CLI cold start benchmark (< 2s) +- Pipeline performance (CR < 2min, import < 5min) +- Fault injection tests (--simulate-failure) +- Purge 24h validation (requires time manipulation) + +**Why defer to weekly:** Benchmarks need stable environment, fault injection requires special setup + +--- + +## QA Effort Estimate + +**QA test development effort only:** + +| Priority | Count | Effort Range | Notes | +|----------|-------|-------------|-------| +| P0 | ~11 | ~2-3 weeks | Complex setup (PII canary, fallback chains, security) | +| P1 | ~31 | ~3-5 weeks | Standard unit/integration (config, storage, connectors, core) | +| P2 | ~5 | ~2-4 days | Edge cases, benchmarks | +| P3 | ~1 | ~0.5-1 day | Single benchmark | +| **Total** | **~48** | **~6-9 weeks** | **1 QA engineer, full-time** | + +**Assumptions:** + +- Includes test design, implementation, debugging, CI integration +- Excludes ongoing maintenance (~10% effort) +- Assumes test infrastructure (trait mocks, test helpers) ready from Sprint 0 +- P0 effort front-loaded in Sprint 0; P1 spread across Sprint 0-2 + +--- + +## Appendix A: Code Examples & Tagging + +**Rust Test Tagging for Selective Execution:** + +```rust +// P0 critical test - security +#[test] +fn anonymize_masks_email_patterns() { + let input = "Contact user@example.com for details"; + let result = anonymize(input); + assert!(!result.contains("user@example.com")); + assert!(result.contains("[EMAIL_REDACTED]")); +} + +// P0 critical test - fallback +#[test] +fn jira_unavailable_triggers_csv_fallback() { + let mock_client = MockJiraClient::new(); + mock_client.expect_import().returning(|_| Err(ConnectorError::Unavailable)); + + let result = import_tickets(&mock_client, &csv_fallback_path); + assert!(result.is_ok()); + assert_eq!(result.unwrap().source, DataSource::CsvFallback); +} + +// P1 integration test - pipeline +#[test] +fn triage_pipeline_produces_scored_tickets() { + let db = TestDb::new_temp(); + let config = test_config(); + let mock_jira = MockJiraClient::with_fixtures("tests/fixtures/tickets.json"); + + let result = run_triage_pipeline(&config, &mock_jira, &db); + assert!(result.is_ok()); + + let scored = result.unwrap(); + assert!(scored.iter().all(|t| t.score.is_some())); + assert!(scored.iter().any(|t| t.score == Some(Score::Go))); +} +``` + +**Run tests by scope:** + +```bash +# All tests (PR default, <10 min) +cargo test + +# Only unit tests (fast feedback) +cargo test --lib + +# Only integration tests +cargo test --test '*' + +# Specific crate +cargo test -p tf-security + +# Ignored tests (nightly/weekly benchmarks) +cargo test -- --ignored + +# With output for debugging +cargo test -- --nocapture +``` + +--- + +## Appendix B: Knowledge Base References + +- **Risk Governance**: `risk-governance.md` - Risk scoring methodology (probability x impact, 1-9) +- **Test Priorities Matrix**: `test-priorities-matrix.md` - P0-P3 criteria and decision tree +- **Test Levels Framework**: `test-levels-framework.md` - Unit vs Integration vs E2E selection +- **Test Quality**: `test-quality.md` - Definition of Done (deterministic, isolated, <300 lines, <1.5 min) +- **ADR Quality Readiness**: `adr-quality-readiness-checklist.md` - 8-category 29-criteria NFR assessment + +--- + +**Generated by:** BMad TEA Agent +**Workflow:** `_bmad/tea/testarch/test-design` +**Version:** 5.0 (BMad v6)