diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md new file mode 100644 index 00000000..9bf87be8 --- /dev/null +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -0,0 +1,178 @@ +--- +sidebar_position: 4 +--- + +# Web3 Adapter + +The Web3 Adapter is the bridge between a platform's local database and eVault. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. + +Web3 Adapter is only needed when you have a database which you want to +automatically sync with eVaults, if your application is stateless or the application only writes to the eVault directly then you don't need a Web3 Adapter. + +## Overview + +Platforms keep their own schemas and databases. To participate in W3DS, they need a component that: + +- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's and/or target eVault (via the user's [eName](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). +- **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. See [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for more details. + +Please Note: That neither inbound or outbound are immediate, or have any transactional guarantees. + +The Web3 Adapter implements this bridge. Understanding its core ideas helps you design better ontologies, mappings, and consistency strategies. + +Please note that the web3 adapter may not come pre-assembled with hooks for all databases. + +### Key Features + +- **Bidirectional mapping**: Local schema ↔ global ontology via JSON mapping configs. +- **ID mapping**: Stores pairs of (localId, globalId) so the same entity is recognized across sync and webhooks. When using our implementation of the Web3 Adapter, you don't need to worry about ID Mapping yourself, the adapter already handles it. +- **eVault client**: Resolves [eNames](/docs/W3DS%20Basics/W3ID) via the Registry, obtains platform tokens, and calls eVault GraphQL (store/update) with retries and health checks. +- **Change handling**: `handleChange` is the main entry for outbound sync; webhook handlers use `fromGlobal` for inbound. + +## Theory: Core Ideas + +### 1. Universal ontology as the common language + +All platforms agree on a small set of global schemas (e.g. User, Post, Group) identified by W3IDs. Each platform maps its local tables to these schemas. The ontology is the contract: if you store data in that shape in an eVault, other platforms can interpret it via their own mappings. Better ontology design (versioning, optional fields, extensibility) leads to easier evolution and fewer breaking changes. + +### 2. Per-entity owner eVault (W3ID) + +Every piece of data has an "owner" — the [eName](/docs/W3DS%20Basics/W3ID) of the eVault where it lives. The adapter uses `ownerEnamePath` in the mapping to determine the owner from the local entity (e.g. a direct field `ename` or a nested path like `user(createdBy.ename)`). Data is always written to that owner's eVault. This keeps ownership explicit and supports ACLs and multi-tenant resolution. + +### 3. Bidirectional mapping and ID mapping + +- **Field mapping**: `localToUniversalMap` defines how each local field maps to a global field (including relations and special functions like `__date`, `__calc`). The same map is used in both directions: `toGlobal` for outbound, `fromGlobal` for inbound. +- **ID mapping**: A separate store (e.g. SQLite `MappingDatabase`) holds `(globalId, localId)`. When syncing out, after a successful `storeMetaEnvelope` the adapter stores the new global ID against the local ID. When a webhook arrives, the adapter looks up the global ID to decide whether to create or update the local entity and then stores or updates the mapping. Without this, the same logical entity could be duplicated or never linked across platforms. When consuming our TypeScript implementation of the Web3 Adapter, this is already taken care of. + +### 4. Change detection on the platform side + +The adapter does not poll the database. The platform must detect changes (e.g. via ORM hooks, DB triggers, or application events) and call `handleChange({ data, tableName, participants })`. So the core idea is: the platform owns the trigger; the adapter owns the translation and eVault write. Better detection (e.g. transactional outbox, CDC) can improve consistency and avoid missed or duplicate syncs. + +### 5. Webhooks (Awareness Protocol) propagate to other platforms + +After data is stored or updated in an eVault, the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) delivers webhooks to all other registered platforms. Those platforms use the same adapter's `fromGlobal` and ID mapping to apply the change locally. So the full loop is: Platform A → adapter → eVault → Awareness Protocol → Platform B's webhook → adapter → Platform B's DB. + +### Design Limitations + +Current implementation has the following known limitations, which we aim to fix with subsequent versions: + +- **Ontology design**: Clear schema versioning, optional vs required fields, and conventions for references (e.g. eNames vs local IDs in payloads). +- **Mapping expressiveness**: Richer `ownerEnamePath` (e.g. fallbacks), relation resolution, and handling of arrays and nested structures. +- **Conflict handling**: The current implementation is last-write-wins via eVault updates; you could add version fields, conflict resolution, or merge strategies. +- **Eventual consistency**: The system does not guarantee ordering or at-least-once delivery of webhooks; you could add idempotency keys, retries, or acknowledgments. + +## Architecture + +```mermaid +graph TB + subgraph Platform["Platform"] + App[Application] + DB[(Local DB)] + WebhookHandler[Webhook Handler
POST /api/webhook] + end + + subgraph Web3Adapter["Web3 Adapter"] + Adapter[Web3Adapter
handleChange / fromGlobal] + Mapper[Mapper
toGlobal / fromGlobal] + MappingDB[(MappingDatabase
localId ↔ globalId)] + EVaultClient[EVaultClient] + end + + subgraph External["External"] + Registry[Registry
resolve eName] + EVault[eVault Core
GraphQL] + end + + App -->|Entity change| Adapter + Adapter --> Mapper + Adapter --> MappingDB + Adapter --> EVaultClient + EVaultClient -->|GET /resolve| Registry + EVaultClient -->|storeMetaEnvelope
updateMetaEnvelopeById| EVault + EVault -->|Awareness Protocol| WebhookHandler + WebhookHandler --> Adapter + Adapter --> DB +``` + +## Implementation + +### Components + +- **Web3Adapter** (`infrastructure/web3-adapter`): Main class. Loads mapping configs from JSON files, owns `MappingDatabase` and `EVaultClient`, and exposes `handleChange` and `fromGlobal`. Config includes `schemasPath`, `dbPath`, `registryUrl`, `platform` (name used for platform token). +- **EVaultClient** (`src/evault/evault.ts`): Resolves eName to GraphQL endpoint via Registry `GET /resolve?w3id=`, obtains platform token via `POST /platforms/certification`, caches clients per eName, and performs health checks (`HEAD /whois`). Uses retries and timeouts for store/update/fetch. +- **Mapper** (`src/mapper/mapper.ts`): `toGlobal({ data, mapping, mappingStore })` and `fromGlobal(...)` using `IMapping` and `MappingDatabase` for resolving relation IDs. +- **MappingDatabase** (`src/db/mapping.db.ts`): SQLite store for `(local_id, global_id)`. Methods: `storeMapping`, `getLocalId(globalId)`, `getGlobalId(localId)`. + +### Flow: Outbound (local change → eVault) + +1. Platform detects a change and calls `adapter.handleChange({ data, tableName, participants })`. +2. Adapter loads the mapping for `tableName`; if missing or `readOnly`, returns. +3. If a global ID already exists for this local ID, adapter calls `toGlobal`, then `evaultClient.updateMetaEnvelopeById(existingGlobalId, { ... })` (fire-and-forget). Optionally uses `lockedIds` to avoid re-entrant sync from webhooks. +4. If no global ID yet, adapter calls `toGlobal` to get owner eName and global-shaped payload. If no owner, returns. Otherwise calls `evaultClient.storeMetaEnvelope({ id: null, w3id, data, schemaId })`, then `mappingDb.storeMapping({ localId, globalId })`. If `participants` includes other eNames, adapter may call `storeReference(ownerEvault/globalId, otherEvault)` for each. +5. [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) later delivers webhooks to other platforms; the originating platform is excluded. + +### EVaultClient details + +- **Resolution**: `GET /resolve?w3id=@...` → response `{ uri }` → GraphQL endpoint is `uri + "/graphql"`. +- **Caching**: One GraphQL client per eName; if health check fails (e.g. `HEAD /whois`), client is evicted and re-resolved next time. +- **Retries**: Configurable for store/update/fetch; typically no retry on 4xx. + +### Mapping configuration (IMapping) + +Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the **Web3 Adapter Mapping Rules** in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. + +Each mapping is a JSON file with: + +- **tableName**: Local table (or entity) name. +- **schemaId**: Global ontology UUID. +- **ownerEnamePath**: Path to the owner eName in the local entity (e.g. `"ename"` or `"users(createdBy.ename)"`). Supports fallbacks with `||`. +- **localToUniversalMap**: Object mapping local field names to global field names or expressions (e.g. `"createdAt": "__date(createdAt)"`, relation syntax `"tableName(path),globalAlias"`). +- **readOnly** (optional): If true, `handleChange` does not sync this table to the eVault. + +### Receiving data (inbound) + +When a platform receives an awareness protocol packet at `POST /api/webhook`: + +1. Parse body: `id`, `w3id`, `schemaId`, `data`. +2. Find the mapping whose `schemaId` matches (or map by schema UUID to table name). +3. Call `adapter.fromGlobal({ data: body.data, mapping })` to get local-shaped data. +4. Call `mappingDb.getLocalId(body.id)`; if found, update the existing local entity; otherwise create and then `mappingDb.storeMapping({ localId: newEntity.id, globalId: body.id })`. +5. Return 200. + +See the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) for a full implementation example and the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for the packet format and delivery mechanism. + +## Sequence: Platform → Adapter → eVault + +```mermaid +sequenceDiagram + participant App as Platform App + participant Adapter as Web3 Adapter + participant MappingDB as MappingDatabase + participant EVaultClient as EVaultClient + participant Registry as Registry + participant EVault as eVault Core + + App->>Adapter: handleChange(data, tableName) + Adapter->>MappingDB: getGlobalId(localId) + alt Existing global ID + Adapter->>Adapter: toGlobal(data, mapping) + Adapter->>EVaultClient: updateMetaEnvelopeById(globalId, ...) + EVaultClient->>Registry: resolve eName + EVaultClient->>EVault: GraphQL updateMetaEnvelopeById + else New entity + Adapter->>Adapter: toGlobal(data, mapping) + Adapter->>EVaultClient: storeMetaEnvelope(w3id, data, schemaId) + EVaultClient->>Registry: resolve eName + EVaultClient->>EVault: GraphQL storeMetaEnvelope + EVault-->>EVaultClient: metaEnvelope.id + EVaultClient-->>Adapter: globalId + Adapter->>MappingDB: storeMapping(localId, globalId) + end +``` + +## Limitations & Planned Extensions + +- **Ontology versioning**: Today `schemaId` is a single UUID, there are plans for adding a `schemaVersion` key to allow for schema versioning. +- **Conflict resolution**: Add version or timestamp fields and resolve conflicts in the adapter or in a separate service. +- **Idempotency**: Use idempotency keys in payloads or in the mapping DB to make webhook handling and outbound sync idempotent. +- **Transactional outbox**: Have the platform write changes to an outbox table and have a worker call `handleChange` so that sync is tied to the same transaction as the local write. diff --git a/docs/docs/W3DS Basics/W3ID.md b/docs/docs/W3DS Basics/W3ID.md new file mode 100644 index 00000000..d47b9cb4 --- /dev/null +++ b/docs/docs/W3DS Basics/W3ID.md @@ -0,0 +1,100 @@ +--- +sidebar_position: 2 +--- + +# W3ID + +W3ID (Web 3 Identifier) is the main identifier for the whole W3DS ecosystem. W3IDs are UUID-based, persistent, and globally unique. When the term **eName** is used, it means a universally resolvable W3ID—one that can be resolved via the Registry to an eVault (or service) endpoint. + +## Overview + +In W3DS, every user, group, eVault, and many objects are identified by a **W3ID**. The same identifier is used across platforms, eVaults, and the Registry. An **eName** is a W3ID that is registered in the Registry and can therefore be resolved to a concrete service URL (e.g. an eVault). So: **eName = W3ID + registered in the Registry**. + +### Key Concepts + +- **W3ID**: The primary identifier for entities in the ecosystem; UUID-based (see [RFC 4122](https://datatracker.ietf.org/doc/html/rfc4122)). +- **eName**: A **universally resolvable** W3ID. Resolving an eName via the Registry yields the eVault (or controller) URL for that identity. +- **Global vs local**: Global IDs (e.g. `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a`) are the primary persistent identity. Local IDs (e.g. `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a/f2a6743e-8d5b-43bc-a9f0-1c7a3b9e90d7`) refer to an object *within* an eVault: the part after the slash is the object UUID in the context of the eVault identified by the part before the slash. + +## W3ID Format + +The W3ID URI format is: + +- **Global**: `@` (case insensitive). The number and positioning of dashes follow RFC 4122. Example: `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a` +- **Local**: `@/` — the object `object-UUID` at the eVault (or owner) `eVault-UUID`. Example: `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a/f2a6743e-8d5b-43bc-a9f0-1c7a3b9e90d7` + +The UUID namespace has range 2^122, which is far larger than the expected number of identities (e.g. 10^22), so collision risk is negligible. + +## Registry Resolution (eName) + +What makes a W3ID an **eName** is that it is registered in the Registry and can be resolved to a service URL: + +1. A client sends `GET /resolve?w3id=@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a` to the Registry. +2. The Registry returns the eVault (or controller) URL for that W3ID. +3. The client can then call that URL (e.g. `/graphql`, `/whois`) with the eName in the `X-ENAME` header. + +So **eName** means: a W3ID that is universally resolvable via the Registry. Users and groups typically have eNames; internal or local-only identifiers may be W3IDs that are not registered and thus not eNames. + +## Where W3IDs Appear + +```mermaid +graph LR + subgraph entities [Entities] + Users[Users] + Groups[Groups] + EVaults[eVaults] + MetaEnvelopes[MetaEnvelopes] + end + + subgraph usage [Usage] + ACLs[ACLs] + KeyBinding[Key Binding] + Owner[Owner eVault] + end + + Users --> W3ID[W3ID / eName] + Groups --> W3ID + EVaults --> W3ID + MetaEnvelopes --> Owner + Owner --> W3ID + ACLs --> W3ID + KeyBinding --> W3ID +``` + +- **Users and groups**: Each has a persistent W3ID (typically an eName). For a person, the W3ID is the long-lived anchor that connects keys (eID certificate, PKI) and the physical person (body characteristics, passport, friends). +- **eVaults**: An eVault may have its own internal W3ID used for syncing between clones or by the hosting provider; the user’s eName is what identifies the “owner” of the vault. +- **MetaEnvelopes**: Each envelope has an owner (a W3ID/eName) and an optional global ID; the W3ID URI scheme can be used to refer to an envelope (e.g. `@/`). +- **ACLs**: Access control lists reference W3IDs (eNames) to indicate who can access data. +- **Key binding**: Public keys in the eVault are bound to the user’s W3ID (eName) via key binding certificates. + +## Key Binding and Recovery + +The identifier is **loosely bound** to a set of keys: the W3ID is not derived from the keys. That allows: + +- **Key rotation**: Keys can be changed (e.g. after compromise or device loss) without changing the W3ID. +- **Friend-based recovery**: A trust list (e.g. 2–3 friends or notaries) can verify identity and approve key changes. The user defines this list while they still have access to their keys. +- **eVault migration**: When a user migrates from one eVault to another, the Registry can store also-known-as (redirect) records so that resolution of the same eName continues to work (e.g. requests for the old eVault W3ID are redirected to the new eVault). + +## Document Binding + +The identifier can be loosely bound to a passport via a binding document certified by a root CA, where the identifier is connected to entropy derived from passport details. Passport verification itself is out of scope for W3ID and is handled by the eID Wallet application. + +## Technical Requirements and Guarantees + +- The identifier must be **globally persistent** and **unique**. +- The identifier must live in a namespace with range greater than 10^22. +- The identifier must support **rotation of secrets** and must be only **loosely bound** to keys. +- The identifier may be loosely tied to a binding document (e.g. passport). + +## Implementation + +The W3ID system is implemented in the `w3id` package (TypeScript) and provides: + +- **W3IDBuilder**: Builder pattern for creating W3IDs (with entropy, namespace, global/local, signer, repository, next-key hash). +- **ID log manager**: Immutable, signed event logs for key rotation and identity updates. +- **JWT signing**: A W3ID with a signer can sign JWTs (e.g. for authentication or key binding certificates). + +This package is useful to create W3IDs with keys or make them global, it is +consumed currently by [eID Wallet](/docs/Infrastructure/eID-Wallet) and [Web3 Adapter](/docs/Infrastructure/Web3-Adapter) + +For implementation details (builder API, storage backends, logging format), see the `w3id` package in the repository. diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md new file mode 100644 index 00000000..0a94a985 --- /dev/null +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -0,0 +1,116 @@ +--- +sidebar_position: 4 +--- + +# Awareness Protocol + +:::warning Prototype-level implementation +The Awareness Protocol described here is **prototype-level**. The packet format, delivery behavior, and platform contract are subject to change in upcoming updates +::: + +The Awareness Protocol is the webhook delivery mechanism in W3DS. When data in an eVault changes (create or update), the eVault notifies every registered platforms so they can stay in sync. "Awareness" means platforms become aware of changes that happened elsewhere. + +## Overview + +Platforms do not poll eVaults for changes. Instead, eVault Core pushes change notifications to every registered platform (except the one that originated the change) via HTTP POST to each platform's `/api/webhook` endpoint. The payload is called an **awareness protocol packet**. Platforms use this packet to apply the same change locally (e.g. create or update an entity in their database) and to maintain ID mappings between global and local IDs. + +### Key Properties + +- **Push-based**: eVault initiates delivery; platforms do not poll. +- **Fire-and-forget**: The eVault does not wait for platforms to acknowledge; failures are logged but do not block the store/update operation. +- **No retries**: There are no automatic retries for failed webhook deliveries in the current implementation. +- **Requestor excluded**: The platform that made the GraphQL request (store/update) is excluded from the list of recipients to avoid "webhook ping-pong." + +## When the Protocol Runs + +The Awareness Protocol is triggered after: + +1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay** to ensure the requesting platform can be reliably identified and excluded from recipients, preventing the same platform from receiving its own write-back and creating a feedback loop ("webhook ping-pong"). +2. **updateMetaEnvelopeById** (update): After the MetaEnvelope is updated, webhooks are sent **immediately** (fire-and-forget, no delay). + +## Mechanism + +```mermaid +sequenceDiagram + participant PlatformA as Platform A + participant EVault as eVault Core + participant Registry as Registry + participant PlatformB as Platform B + participant PlatformC as Platform C + + PlatformA->>EVault: storeMetaEnvelope / updateMetaEnvelopeById + EVault->>EVault: Persist to Neo4j + alt storeMetaEnvelope + Note over EVault: Wait 3 seconds + end + EVault->>Registry: GET /platforms + Registry-->>EVault: List of platform base URLs + EVault->>EVault: Filter out requesting platform + EVault->>PlatformB: POST /api/webhook (payload) + EVault->>PlatformC: POST /api/webhook (payload) + Note over EVault,PlatformC: All requests in parallel, 5s timeout each, no retries +``` + +### Step-by-Step + +1. **Platform list**: eVault calls the Registry with `GET /platforms` and receives a list of platform base URLs (e.g. `["https://blabsy.example.com", "https://pictique.example.com", ...]`). +2. **Filter**: The requesting platform is removed from the list. The requestor is identified from the Bearer token's `platform` claim (the platform URL that was certified when the token was issued). URL comparison is normalized so that equivalent URLs are treated as the same. +3. **Delivery**: For each remaining URL, the eVault sends `POST {platformUrl}/api/webhook` with the awareness protocol payload. Each request has a 5-second timeout. Requests are sent in parallel (`Promise.allSettled`); if one fails, others still run, and failures are logged without blocking the mutation. + +## Packet Format (Awareness Protocol Payload) + +The body of each webhook request is JSON with the following fields: + +| Field | Description | +|-------|-------------| +| `id` | MetaEnvelope ID (W3ID). | +| `w3id` | Owner eName (eVault owner W3ID). | +| `schemaId` | Ontology/schema UUID (identifies the type of entity and which mapping the platform should use). | +| `data` | The entity payload in the **global ontology** shape (parsed key-value structure). | + +In the current version of the implementation the entire payload is sent in +plain text to any registered platform, so all data is sent to every platform and +it's the platform's responsibility to reject any packets it doesn't use, in +future versions of awareness protocol, it will be changed so that platforms can +subscribe to certain ontology changes and they will be provided details of the +updated MetaEnvelope by reference instead of value. + +**Content-Type**: `application/json` + +**Example**: + +```json +{ + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "w3id": "@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a", + "schemaId": "550e8400-e29b-41d4-a716-446655440001", + "data": { + "content": "Hello, world!", + "mediaUrls": [], + "authorId": "@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a", + "createdAt": "2025-01-24T10:00:00Z" + } +} +``` + +## Platform Contract + +Platforms that participate in W3DS must implement an HTTP endpoint that accepts awareness protocol packets: + +- **Method and path**: `POST /api/webhook` +- **Request**: JSON body as described above. +- **Behavior**: The platform should (1) use `schemaId` to find the correct mapping from global ontology to local schema, (2) transform `data` from global to local format (e.g. using the Web3 Adapter's `fromGlobal`), (3) resolve or create the local entity and store the global-ID-to-local-ID mapping, (4) return HTTP 200 on success. +- **Idempotency**: Implementors are encouraged to treat the same `id` (global ID) as idempotent (create or update the same local entity) so that duplicate or retried deliveries do not create duplicates. + +For a step-by-step implementation guide, see the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) in the Post Platform Guide. + +## Limitations and Extension Points + +The current protocol has limitations which are going to be improved in subsequent versions: + +- **No retries**: Failed webhooks are not retried. A more robust system could add retries with backoff or a dead-letter queue. +- **No ordering guarantee**: Webhooks to different platforms are sent in parallel; there is no guarantee of order across platforms or across multiple mutations. +- **No at-least-once guarantee**: Because delivery is fire-and-forget and there are no retries, a platform might never receive a given update. At-least-once delivery would require acknowledgments and retries (and possibly idempotency keys). +- **Platform list from Registry**: The set of recipients is whatever the Registry returns for `GET /platforms`. This is a prototype level shortcut and will be phased out. + +Designing retries, ordering, or delivery guarantees would be natural extension points for a production-grade awareness mechanism.