diff --git a/docs/E2E_and_Neo4j_Task_Planning_REVISED.md b/docs/E2E_and_Neo4j_Task_Planning_REVISED.md new file mode 100644 index 0000000..1622b63 --- /dev/null +++ b/docs/E2E_and_Neo4j_Task_Planning_REVISED.md @@ -0,0 +1,33 @@ +# E2E and Neo4j Task Planning (Revised — Interpreter Terminology) + +This plan aligns E2E testing and the Neo4j refactor with the Interpreter Management System and current API contracts. + +## Story: E2E Testing & Neo4j Integration +- ID: story:e2e-testing +- Objective: Establish reliable E2E scaffolding (pytest + Playwright) to validate SciDK core flows and support Neo4j persistence refactor. + +## Phases +1. Smoke E2E baseline: Validate core flows (Scan, Browse, Interpreters, Map) without Neo4j. +2. Neo4j refactor: Make Neo4j the live graph store (foundational). +3. Expanded E2E: Add Neo4j-specific tests, interpreter workflows, and negatives. + +## Success Criteria +- Core MVP flows pass E2E in CI; Neo4j driver integration solid and tested. +- Interpreter registration and execution validated E2E. + +## Tasks +- task:e2e:01-smoke-baseline — Playwright smoke E2E baseline (MVP flows). RICE 999. Status: Ready. +- task:e2e:02-neo4j-refactor — Neo4j as live graph store. RICE 998. Status: Ready. +- task:e2e:03-expanded-e2e — Neo4j-specific E2E + interpreter workflows + negatives. RICE 997. Status: Planned. + +## Interpreter Terminology and APIs +- Use Interpreters (not Enrichers) consistently. +- Required endpoints: GET/POST /api/interpreters, GET /api/interpreters/, POST /api/interpreters//test, POST /api/scans//interpret. + +## E2E Notes +- Prefer BASE_URL injection; keep smoke tests fast (<5s/spec) and independent of external services. +- Add data-testid hooks for Settings, Interpreters, Map, Scan flows. + +## References +- MVP_Architecture_Overview_REVISED.md +- SciDK_Interpreter_Management_System.md diff --git a/docs/MVP_Architecture_Overview_REVISED.md b/docs/MVP_Architecture_Overview_REVISED.md new file mode 100644 index 0000000..de2e270 --- /dev/null +++ b/docs/MVP_Architecture_Overview_REVISED.md @@ -0,0 +1,83 @@ +# MVP Architecture Overview (Revised — Interpreter‑centric) + +This document aligns the MVP architecture with the Interpreter Management System and current repository terminology. Interpreters are lightweight, read‑only metadata extractors that understand specific file formats. + +## Core UI Areas +- Home / Scan: start scans via POST /api/scan (or background via /api/tasks) +- Files / Browse: explore scan snapshot via GET /api/scans//fs +- Interpreters: render per‑file insights (Python, CSV, IPYNB for MVP) +- Map: view schema and export instances +- Interpreter Settings: configure interpreter assignments/rules and registration +- Rclone Mounts (feature‑flagged): manage safe local FUSE mounts +- Background Tasks: monitor async scan/interpret/commit + +## Key APIs (MVP) + +### Filesystem providers +- GET /api/providers +- GET /api/provider_roots?provider_id= +- GET /api/browse?provider_id=&root_id=&path=[&recursive=false&max_depth=1&fast_list=false] +- POST /api/scan +- GET /api/datasets, GET /api/datasets/ + +### Scans +- GET /api/scans//status +- GET /api/scans//fs +- POST /api/scans//interpret + - Body: { include?, exclude?, max_size_bytes?, after_rowid?, max_files?, overwrite? } + - Returns: { status, processed_count, error_count, filtered_by_size, filtered_by_include, filtered_no_interpreter, next_cursor } +- POST /api/scans//commit + - Returns commit summary including optional Neo4j verification fields + +### Background tasks +- POST /api/tasks { type: 'scan' | 'commit' | 'interpret', ... } +- GET /api/tasks, GET /api/tasks/ + +### Interpreters: registry and execution +- GET /api/interpreters → list available interpreters { id, name, runtime, supported_extensions, metadata_schema } +- GET /api/interpreters/ +- POST /api/interpreters → register new interpreter { name, runtime, extensions, script, metadata_schema, ... } +- POST /api/interpreters//test → run test on a sample file { file_path } → { status, result, errors, warnings, execution_time_ms } + +### Graph: schema and instance exports +- GET /api/graph/schema +- GET /api/graph/schema.neo4j (optional; 501 if driver/misconfig) +- GET /api/graph/schema.apoc (optional; 502 if APOC unavailable) +- GET /api/graph/instances.csv?label=