From af574b6611182ac06f2fa3704949d1fc46569cd4 Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Thu, 29 Jan 2026 18:02:21 +0530 Subject: [PATCH 01/10] docs: add docs for w3id, w3 adapter and awareness protocol --- docs/docs/Infrastructure/Web3-Adapter.md | 174 ++++++++++++++++++ docs/docs/W3DS Basics/W3ID.md | 97 ++++++++++ docs/docs/W3DS Protocol/Awareness-Protocol.md | 111 +++++++++++ 3 files changed, 382 insertions(+) create mode 100644 docs/docs/Infrastructure/Web3-Adapter.md create mode 100644 docs/docs/W3DS Basics/W3ID.md create mode 100644 docs/docs/W3DS Protocol/Awareness-Protocol.md diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md new file mode 100644 index 00000000..12a77a76 --- /dev/null +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -0,0 +1,174 @@ +--- +sidebar_position: 4 +--- + +# Web3 Adapter + +The Web3 Adapter is the bridge between a platform's local database and the W3DS global ontology. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. + +## Overview + +Platforms keep their own schemas and databases. To participate in W3DS, they need a component that: + +- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's eVault (via [W3ID](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). +- **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. + +The Web3 Adapter implements this bridge. Understanding its core ideas helps you design better ontologies, mappings, and consistency strategies. + +### Key Features + +- **Bidirectional mapping**: Local schema ↔ global ontology via JSON mapping configs. +- **ID mapping**: Stores pairs of (localId, globalId) so the same entity is recognized across sync and webhooks. +- **eVault client**: Resolves [eNames](/docs/W3DS%20Basics/W3ID) via the Registry, obtains platform tokens, and calls eVault GraphQL (store/update) with retries and health checks. +- **Change handling**: `handleChange` is the main entry for outbound sync; webhook handlers use `fromGlobal` for inbound. + +## Theory: Core Ideas + +### 1. Universal ontology as the common language + +All platforms agree on a small set of global schemas (e.g. User, Post, Group) identified by UUIDs. Each platform maps its local tables to these schemas. The ontology is the contract: if you store data in that shape in an eVault, other platforms can interpret it via their own mappings. Better ontology design (versioning, optional fields, extensibility) leads to easier evolution and fewer breaking changes. + +### 2. Per-entity owner eVault (W3ID) + +Every piece of data has an "owner" — the [eName](/docs/W3DS%20Basics/W3ID) of the eVault where it lives. The adapter uses `ownerEnamePath` in the mapping to determine the owner from the local entity (e.g. a direct field `ename` or a nested path like `user(createdBy.ename)`). Data is always written to that owner's eVault. This keeps ownership explicit and supports ACLs and multi-tenant resolution. + +### 3. Bidirectional mapping and ID mapping + +- **Field mapping**: `localToUniversalMap` defines how each local field maps to a global field (including relations and special functions like `__date`, `__calc`). The same map is used in both directions: `toGlobal` for outbound, `fromGlobal` for inbound. +- **ID mapping**: A separate store (e.g. SQLite `MappingDatabase`) holds `(globalId, localId)`. When syncing out, after a successful `storeMetaEnvelope` the adapter stores the new global ID against the local ID. When a webhook arrives, the adapter looks up the global ID to decide whether to create or update the local entity and then stores or updates the mapping. Without this, the same logical entity could be duplicated or never linked across platforms. + +### 4. Change detection on the platform side + +The adapter does not poll the database. The platform must detect changes (e.g. via ORM hooks, DB triggers, or application events) and call `handleChange({ data, tableName, participants })`. So the core idea is: the platform owns the trigger; the adapter owns the translation and eVault write. Better detection (e.g. transactional outbox, CDC) can improve consistency and avoid missed or duplicate syncs. + +### 5. Webhooks (Awareness Protocol) propagate to other platforms + +After data is stored or updated in an eVault, the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) delivers webhooks to all other registered platforms. Those platforms use the same adapter's `fromGlobal` and ID mapping to apply the change locally. So the full loop is: Platform A → adapter → eVault → Awareness Protocol → Platform B's webhook → adapter → Platform B's DB. + +### Design implications + +You can get better results by: + +- **Ontology design**: Clear schema versioning, optional vs required fields, and conventions for references (e.g. eNames vs local IDs in payloads). +- **Mapping expressiveness**: Richer `ownerEnamePath` (e.g. fallbacks), relation resolution, and handling of arrays and nested structures. +- **Conflict handling**: The current implementation is last-write-wins via eVault updates; you could add version fields, conflict resolution, or merge strategies. +- **Eventual consistency**: The system does not guarantee ordering or at-least-once delivery of webhooks; you could add idempotency keys, retries, or acknowledgments. + +## Architecture + +```mermaid +graph TB + subgraph Platform["Platform"] + App[Application] + DB[(Local DB)] + WebhookHandler[Webhook Handler
POST /api/webhook] + end + + subgraph Web3Adapter["Web3 Adapter"] + Adapter[Web3Adapter
handleChange / fromGlobal] + Mapper[Mapper
toGlobal / fromGlobal] + MappingDB[(MappingDatabase
localId ↔ globalId)] + EVaultClient[EVaultClient] + end + + subgraph External["External"] + Registry[Registry
resolve eName] + EVault[eVault Core
GraphQL] + end + + App -->|Entity change| Adapter + Adapter --> Mapper + Adapter --> MappingDB + Adapter --> EVaultClient + EVaultClient -->|GET /resolve| Registry + EVaultClient -->|storeMetaEnvelope
updateMetaEnvelopeById| EVault + EVault -->|Awareness Protocol| WebhookHandler + WebhookHandler --> Adapter + Adapter --> DB +``` + +## Implementation + +### Components + +- **Web3Adapter** (`infrastructure/web3-adapter`): Main class. Loads mapping configs from JSON files, owns `MappingDatabase` and `EVaultClient`, and exposes `handleChange` and `fromGlobal`. Config includes `schemasPath`, `dbPath`, `registryUrl`, `platform` (name used for platform token). +- **EVaultClient** (`src/evault/evault.ts`): Resolves eName to GraphQL endpoint via Registry `GET /resolve?w3id=`, obtains platform token via `POST /platforms/certification`, caches clients per eName, and performs health checks (`HEAD /whois`). Uses retries and timeouts for store/update/fetch. +- **Mapper** (`src/mapper/mapper.ts`): `toGlobal({ data, mapping, mappingStore })` and `fromGlobal(...)` using `IMapping` and `MappingDatabase` for resolving relation IDs. +- **MappingDatabase** (`src/db/mapping.db.ts`): SQLite store for `(local_id, global_id)`. Methods: `storeMapping`, `getLocalId(globalId)`, `getGlobalId(localId)`. + +### Flow: Outbound (local change → eVault) + +1. Platform detects a change and calls `adapter.handleChange({ data, tableName, participants })`. +2. Adapter loads the mapping for `tableName`; if missing or `readOnly`, returns. +3. If a global ID already exists for this local ID, adapter calls `toGlobal`, then `evaultClient.updateMetaEnvelopeById(existingGlobalId, { ... })` (fire-and-forget). Optionally uses `lockedIds` to avoid re-entrant sync from webhooks. +4. If no global ID yet, adapter calls `toGlobal` to get owner eName and global-shaped payload. If no owner, returns. Otherwise calls `evaultClient.storeMetaEnvelope({ id: null, w3id, data, schemaId })`, then `mappingDb.storeMapping({ localId, globalId })`. If `participants` includes other eNames, adapter may call `storeReference(ownerEvault/globalId, otherEvault)` for each. +5. [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) later delivers webhooks to other platforms; the originating platform is excluded. + +### EVaultClient details + +- **Resolution**: `GET /resolve?w3id=@...` → response `{ uri }` → GraphQL endpoint is `uri + "/graphql"`. +- **Caching**: One GraphQL client per eName; if health check fails (e.g. `HEAD /whois`), client is evicted and re-resolved next time. +- **Retries**: Configurable for store/update/fetch; typically no retry on 4xx. + +### Mapping configuration (IMapping) + +Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the **Web3 Adapter Mapping Rules** in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. + +Each mapping is a JSON file with: + +- **tableName**: Local table (or entity) name. +- **schemaId**: Global ontology UUID. +- **ownerEnamePath**: Path to the owner eName in the local entity (e.g. `"ename"` or `"users(createdBy.ename)"`). Supports fallbacks with `||`. +- **localToUniversalMap**: Object mapping local field names to global field names or expressions (e.g. `"createdAt": "__date(createdAt)"`, relation syntax `"tableName(path),globalAlias"`). +- **readOnly** (optional): If true, `handleChange` does not sync this table to the eVault. + +For full syntax (direct, relation, array, `__date`, `__calc`, owner path), see the mapping rules in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. + +### Receiving data (inbound) + +When a platform receives an awareness protocol packet at `POST /api/webhook`: + +1. Parse body: `id`, `w3id`, `schemaId`, `data`. +2. Find the mapping whose `schemaId` matches (or map by schema UUID to table name). +3. Call `adapter.fromGlobal({ data: body.data, mapping })` to get local-shaped data. +4. Call `mappingDb.getLocalId(body.id)`; if found, update the existing local entity; otherwise create and then `mappingDb.storeMapping({ localId: newEntity.id, globalId: body.id })`. +5. Return 200. + +See the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) for a full implementation example. + +## Sequence: Platform → Adapter → eVault + +```mermaid +sequenceDiagram + participant App as Platform App + participant Adapter as Web3 Adapter + participant MappingDB as MappingDatabase + participant EVaultClient as EVaultClient + participant Registry as Registry + participant EVault as eVault Core + + App->>Adapter: handleChange(data, tableName) + Adapter->>MappingDB: getGlobalId(localId) + alt Existing global ID + Adapter->>Adapter: toGlobal(data, mapping) + Adapter->>EVaultClient: updateMetaEnvelopeById(globalId, ...) + EVaultClient->>Registry: resolve eName + EVaultClient->>EVault: GraphQL updateMetaEnvelopeById + else New entity + Adapter->>Adapter: toGlobal(data, mapping) + Adapter->>EVaultClient: storeMetaEnvelope(w3id, data, schemaId) + EVaultClient->>Registry: resolve eName + EVaultClient->>EVault: GraphQL storeMetaEnvelope + EVault-->>EVaultClient: metaEnvelope.id + EVaultClient-->>Adapter: globalId + Adapter->>MappingDB: storeMapping(localId, globalId) + end +``` + +## Extension points + +- **Ontology versioning**: Today `schemaId` is a single UUID; you could version schemas and map multiple local tables to different versions. +- **Conflict resolution**: Add version or timestamp fields and resolve conflicts in the adapter or in a separate service. +- **Idempotency**: Use idempotency keys in payloads or in the mapping DB to make webhook handling and outbound sync idempotent. +- **ID mapping storage**: Replace SQLite with a shared store if multiple instances of the platform need the same mappings. +- **Transactional outbox**: Have the platform write changes to an outbox table and have a worker call `handleChange` so that sync is tied to the same transaction as the local write. diff --git a/docs/docs/W3DS Basics/W3ID.md b/docs/docs/W3DS Basics/W3ID.md new file mode 100644 index 00000000..cc7e5d33 --- /dev/null +++ b/docs/docs/W3DS Basics/W3ID.md @@ -0,0 +1,97 @@ +--- +sidebar_position: 2 +--- + +# W3ID + +W3ID (Web 3 Identifier) is the main identifier for the whole W3DS ecosystem. W3IDs are UUID-based, persistent, and globally unique. When the term **eName** is used, it means a universally resolvable W3ID—one that can be resolved via the Registry to an eVault (or service) endpoint. + +## Overview + +In W3DS, every user, group, eVault, and many objects are identified by a **W3ID**. The same identifier is used across platforms, eVaults, and the Registry. An **eName** is a W3ID that is registered in the Registry and can therefore be resolved to a concrete service URL (e.g. an eVault). So: **eName = W3ID + registered in the Registry**. + +### Key Concepts + +- **W3ID**: The primary identifier for entities in the ecosystem; UUID-based (see [RFC 4122](https://datatracker.ietf.org/doc/html/rfc4122)). +- **eName**: A **universally resolvable** W3ID. Resolving an eName via the Registry yields the eVault (or controller) URL for that identity. +- **Global vs local**: Global IDs (e.g. `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a`) are the primary persistent identity. Local IDs (e.g. `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a/f2a6743e-8d5b-43bc-a9f0-1c7a3b9e90d7`) refer to an object *within* an eVault: the part after the slash is the object UUID in the context of the eVault identified by the part before the slash. + +## W3ID Format + +The W3ID URI format is: + +- **Global**: `@` (case insensitive). The number and positioning of dashes follow RFC 4122. Example: `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a` +- **Local**: `@/` — the object `object-UUID` at the eVault (or owner) `eVault-UUID`. Example: `@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a/f2a6743e-8d5b-43bc-a9f0-1c7a3b9e90d7` + +The UUID namespace has range 2^122, which is far larger than the expected number of identities (e.g. 10^22), so collision risk is negligible. + +## Registry Resolution (eName) + +What makes a W3ID an **eName** is that it is registered in the Registry and can be resolved to a service URL: + +1. A client sends `GET /resolve?w3id=@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a` to the Registry. +2. The Registry returns the eVault (or controller) URL for that W3ID. +3. The client can then call that URL (e.g. `/graphql`, `/whois`) with the eName in the `X-ENAME` header. + +So **eName** means: a W3ID that is universally resolvable via the Registry. Users and groups typically have eNames; internal or local-only identifiers may be W3IDs that are not registered and thus not eNames. + +## Where W3IDs Appear + +```mermaid +graph LR + subgraph entities [Entities] + Users[Users] + Groups[Groups] + EVaults[eVaults] + MetaEnvelopes[MetaEnvelopes] + end + + subgraph usage [Usage] + ACLs[ACLs] + KeyBinding[Key Binding] + Owner[Owner eVault] + end + + Users --> W3ID[W3ID / eName] + Groups --> W3ID + EVaults --> W3ID + MetaEnvelopes --> Owner + Owner --> W3ID + ACLs --> W3ID + KeyBinding --> W3ID +``` + +- **Users and groups**: Each has a persistent W3ID (typically an eName). For a person, the W3ID is the long-lived anchor that connects keys (eID certificate, PKI) and the physical person (body characteristics, passport, friends). +- **eVaults**: An eVault may have its own internal W3ID used for syncing between clones or by the hosting provider; the user’s eName is what identifies the “owner” of the vault. +- **MetaEnvelopes**: Each envelope has an owner (a W3ID/eName) and an optional global ID; the W3ID URI scheme can be used to refer to an envelope (e.g. `@/`). +- **ACLs**: Access control lists reference W3IDs (eNames) to indicate who can access data. +- **Key binding**: Public keys in the eVault are bound to the user’s W3ID (eName) via key binding certificates. + +## Key Binding and Recovery + +The identifier is **loosely bound** to a set of keys: the W3ID is not derived from the keys. That allows: + +- **Key rotation**: Keys can be changed (e.g. after compromise or device loss) without changing the W3ID. +- **Friend-based recovery**: A trust list (e.g. 2–3 friends or notaries) can verify identity and approve key changes. The user defines this list while they still have access to their keys. +- **eVault migration**: When a user migrates from one eVault to another, the Registry can store also-known-as (redirect) records so that resolution of the same eName continues to work (e.g. requests for the old eVault W3ID are redirected to the new eVault). + +## Document Binding + +The identifier can be loosely bound to a passport via a binding document certified by a root CA, where the identifier is connected to entropy derived from passport details. Passport verification itself is out of scope for W3ID and is handled by the eID Wallet application. + +## Technical Requirements and Guarantees + +- The identifier must be **globally persistent** and **unique**. +- The identifier must live in a namespace with range greater than 10^22. +- The identifier must support **rotation of secrets** and must be only **loosely bound** to keys. +- The identifier may be loosely tied to a binding document (e.g. passport). + +## Implementation + +The W3ID system is implemented in the `w3id` package (TypeScript) and provides: + +- **W3IDBuilder**: Builder pattern for creating W3IDs (with entropy, namespace, global/local, signer, repository, next-key hash). +- **ID log manager**: Immutable, signed event logs for key rotation and identity updates. +- **JWT signing**: A W3ID with a signer can sign JWTs (e.g. for authentication or key binding certificates). + +For implementation details (builder API, storage backends, logging format), see the `w3id` package in the repository. diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md new file mode 100644 index 00000000..a13f1a7d --- /dev/null +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -0,0 +1,111 @@ +--- +sidebar_position: 4 +--- + +# Awareness Protocol + +:::warning Prototype-level implementation +The Awareness Protocol described here is **prototype-level**. The packet format, delivery behavior, and platform contract are subject to change. Do not rely on them for production without checking for updates. +::: + +The Awareness Protocol is the webhook deliverability mechanism in W3DS. When data in an eVault changes (create or update), the eVault notifies all other registered platforms so they can stay in sync. "Awareness" means platforms become aware of changes that happened elsewhere. + +## Overview + +Platforms do not poll eVaults for changes. Instead, eVault Core pushes change notifications to every registered platform (except the one that originated the change) via HTTP POST to each platform's `/api/webhook` endpoint. The payload is called an **awareness protocol packet**. Platforms use this packet to apply the same change locally (e.g. create or update an entity in their database) and to maintain ID mappings between global and local IDs. + +### Key Properties + +- **Push-based**: eVault initiates delivery; platforms do not poll. +- **Fire-and-forget**: The eVault does not wait for platforms to acknowledge; failures are logged but do not block the store/update operation. +- **No retries**: There are no automatic retries for failed webhook deliveries in the current implementation. +- **Requestor excluded**: The platform that made the GraphQL request (store/update) is excluded from the list of recipients to avoid "webhook ping-pong." + +## When the Protocol Runs + +The Awareness Protocol is triggered after: + +1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay**. The delay exists because the requesting platform's URL may not be known immediately from the Bearer token; without the delay, the same platform could receive its own write back and create a feedback loop ("webhook ping-pong"). +2. **updateMetaEnvelopeById** (update): After the MetaEnvelope is updated, webhooks are sent **immediately** (fire-and-forget, no delay). + +## Mechanism + +```mermaid +sequenceDiagram + participant PlatformA as Platform A + participant EVault as eVault Core + participant Registry as Registry + participant PlatformB as Platform B + participant PlatformC as Platform C + + PlatformA->>EVault: storeMetaEnvelope / updateMetaEnvelopeById + EVault->>EVault: Persist to Neo4j + alt storeMetaEnvelope + Note over EVault: Wait 3 seconds + end + EVault->>Registry: GET /platforms + Registry-->>EVault: List of platform base URLs + EVault->>EVault: Filter out requesting platform + EVault->>PlatformB: POST /api/webhook (payload) + EVault->>PlatformC: POST /api/webhook (payload) + Note over EVault,PlatformC: All requests in parallel, 5s timeout each, no retries +``` + +### Step-by-Step + +1. **Platform list**: eVault calls the Registry with `GET /platforms` and receives a list of platform base URLs (e.g. `["https://blabsy.example.com", "https://pictique.example.com", ...]`). +2. **Filter**: The requesting platform is removed from the list. The requestor is identified from the Bearer token's `platform` claim (the platform URL that was certified when the token was issued). URL comparison is normalized so that equivalent URLs are treated as the same. +3. **Delivery**: For each remaining URL, the eVault sends `POST {platformUrl}/api/webhook` with the awareness protocol payload. Each request has a 5-second timeout. Requests are sent in parallel (`Promise.allSettled`); if one fails, others still run, and failures are logged without blocking the mutation. + +## Packet Format (Awareness Protocol Payload) + +The body of each webhook request is JSON with the following fields: + +| Field | Description | +|-------|-------------| +| `id` | Global MetaEnvelope ID (UUID). | +| `w3id` | Owner eName (eVault owner W3ID). | +| `schemaId` | Ontology/schema UUID (identifies the type of entity and which mapping the platform should use). | +| `data` | The entity payload in the **global ontology** shape (parsed key-value structure). | +| `evaultPublicKey` | Optional; eVault public key. | + +**Content-Type**: `application/json` + +**Example**: + +```json +{ + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "w3id": "@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a", + "schemaId": "550e8400-e29b-41d4-a716-446655440001", + "data": { + "content": "Hello, world!", + "mediaUrls": [], + "authorId": "@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a", + "createdAt": "2025-01-24T10:00:00Z" + }, + "evaultPublicKey": "z..." +} +``` + +## Platform Contract + +Platforms that participate in W3DS must implement an HTTP endpoint that accepts awareness protocol packets: + +- **Method and path**: `POST /api/webhook` +- **Request**: JSON body as described above. +- **Behavior**: The platform should (1) use `schemaId` to find the correct mapping from global ontology to local schema, (2) transform `data` from global to local format (e.g. using the Web3 Adapter's `fromGlobal`), (3) resolve or create the local entity and store the global-ID-to-local-ID mapping, (4) return HTTP 200 on success. +- **Idempotency**: Implementors are encouraged to treat the same `id` (global ID) as idempotent (create or update the same local entity) so that duplicate or retried deliveries do not create duplicates. + +For a step-by-step implementation guide, see the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) in the Post Platform Guide. + +## Limitations and Extension Points + +The current protocol has intentional limitations that readers may improve in their own implementations: + +- **No retries**: Failed webhooks are not retried. A more robust system could add retries with backoff or a dead-letter queue. +- **No ordering guarantee**: Webhooks to different platforms are sent in parallel; there is no guarantee of order across platforms or across multiple mutations. +- **No at-least-once guarantee**: Because delivery is fire-and-forget and there are no retries, a platform might never receive a given update. At-least-once delivery would require acknowledgments and retries (and possibly idempotency keys). +- **Platform list from Registry**: The set of recipients is whatever the Registry returns for `GET /platforms`. How that list is maintained and updated is registry-specific. + +Designing retries, ordering, or delivery guarantees would be natural extension points for a production-grade awareness mechanism. From 3fac10087a39762d1a3f2891d15662f6377517ab Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Thu, 29 Jan 2026 18:06:18 +0530 Subject: [PATCH 02/10] docs: fix sentence framing --- docs/docs/Infrastructure/Web3-Adapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md index 12a77a76..5eb460f0 100644 --- a/docs/docs/Infrastructure/Web3-Adapter.md +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -4,7 +4,7 @@ sidebar_position: 4 # Web3 Adapter -The Web3 Adapter is the bridge between a platform's local database and the W3DS global ontology. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. +The Web3 Adapter is the bridge between a platform's local database and eVault. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. ## Overview From d6a649cb577a8e9100a61174483bf4e8258696dc Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Thu, 29 Jan 2026 18:09:05 +0530 Subject: [PATCH 03/10] docs: clarify id mapping --- docs/docs/Infrastructure/Web3-Adapter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md index 5eb460f0..bfeb7c69 100644 --- a/docs/docs/Infrastructure/Web3-Adapter.md +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -10,7 +10,7 @@ The Web3 Adapter is the bridge between a platform's local database and eVault. I Platforms keep their own schemas and databases. To participate in W3DS, they need a component that: -- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's eVault (via [W3ID](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). +- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's eVault (via the user's [eName](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). - **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. The Web3 Adapter implements this bridge. Understanding its core ideas helps you design better ontologies, mappings, and consistency strategies. @@ -18,7 +18,7 @@ The Web3 Adapter implements this bridge. Understanding its core ideas helps you ### Key Features - **Bidirectional mapping**: Local schema ↔ global ontology via JSON mapping configs. -- **ID mapping**: Stores pairs of (localId, globalId) so the same entity is recognized across sync and webhooks. +- **ID mapping**: Stores pairs of (localId, globalId) so the same entity is recognized across sync and webhooks. When using our implementation of the Web3 Adapter, you don't need to worry about ID Mapping yourself, the adapter already handles it. - **eVault client**: Resolves [eNames](/docs/W3DS%20Basics/W3ID) via the Registry, obtains platform tokens, and calls eVault GraphQL (store/update) with retries and health checks. - **Change handling**: `handleChange` is the main entry for outbound sync; webhook handlers use `fromGlobal` for inbound. @@ -35,7 +35,7 @@ Every piece of data has an "owner" — the [eName](/docs/W3DS%20Basics/W3ID) of ### 3. Bidirectional mapping and ID mapping - **Field mapping**: `localToUniversalMap` defines how each local field maps to a global field (including relations and special functions like `__date`, `__calc`). The same map is used in both directions: `toGlobal` for outbound, `fromGlobal` for inbound. -- **ID mapping**: A separate store (e.g. SQLite `MappingDatabase`) holds `(globalId, localId)`. When syncing out, after a successful `storeMetaEnvelope` the adapter stores the new global ID against the local ID. When a webhook arrives, the adapter looks up the global ID to decide whether to create or update the local entity and then stores or updates the mapping. Without this, the same logical entity could be duplicated or never linked across platforms. +- **ID mapping**: A separate store (e.g. SQLite `MappingDatabase`) holds `(globalId, localId)`. When syncing out, after a successful `storeMetaEnvelope` the adapter stores the new global ID against the local ID. When a webhook arrives, the adapter looks up the global ID to decide whether to create or update the local entity and then stores or updates the mapping. Without this, the same logical entity could be duplicated or never linked across platforms. When consuming our TypeScript implementation of the Web3 Adapter, this is already taken care of. ### 4. Change detection on the platform side From 6859d8e30161f00cb93655cb9442c05bfbc7e22d Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Thu, 29 Jan 2026 18:16:39 +0530 Subject: [PATCH 04/10] docs: address limitaions --- docs/docs/Infrastructure/Web3-Adapter.md | 15 ++++++--------- docs/docs/W3DS Protocol/Awareness-Protocol.md | 6 +++--- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md index bfeb7c69..8ada7760 100644 --- a/docs/docs/Infrastructure/Web3-Adapter.md +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -45,9 +45,9 @@ The adapter does not poll the database. The platform must detect changes (e.g. v After data is stored or updated in an eVault, the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) delivers webhooks to all other registered platforms. Those platforms use the same adapter's `fromGlobal` and ID mapping to apply the change locally. So the full loop is: Platform A → adapter → eVault → Awareness Protocol → Platform B's webhook → adapter → Platform B's DB. -### Design implications +### Design Limitations -You can get better results by: +Current implementation has the following known limitations, which we aim to fix with subsequent versions: - **Ontology design**: Clear schema versioning, optional vs required fields, and conventions for references (e.g. eNames vs local IDs in payloads). - **Mapping expressiveness**: Richer `ownerEnamePath` (e.g. fallbacks), relation resolution, and handling of arrays and nested structures. @@ -112,7 +112,7 @@ graph TB ### Mapping configuration (IMapping) -Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the **Web3 Adapter Mapping Rules** in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. +Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the [Web3 Adapter Mapping Rules](../../../infrastructure/web3-adapter/MAPPING_RULES.md). Each mapping is a JSON file with: @@ -122,8 +122,6 @@ Each mapping is a JSON file with: - **localToUniversalMap**: Object mapping local field names to global field names or expressions (e.g. `"createdAt": "__date(createdAt)"`, relation syntax `"tableName(path),globalAlias"`). - **readOnly** (optional): If true, `handleChange` does not sync this table to the eVault. -For full syntax (direct, relation, array, `__date`, `__calc`, owner path), see the mapping rules in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. - ### Receiving data (inbound) When a platform receives an awareness protocol packet at `POST /api/webhook`: @@ -134,7 +132,7 @@ When a platform receives an awareness protocol packet at `POST /api/webhook`: 4. Call `mappingDb.getLocalId(body.id)`; if found, update the existing local entity; otherwise create and then `mappingDb.storeMapping({ localId: newEntity.id, globalId: body.id })`. 5. Return 200. -See the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) for a full implementation example. +See the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) for a full implementation example and the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for the packet format and delivery mechanism. ## Sequence: Platform → Adapter → eVault @@ -165,10 +163,9 @@ sequenceDiagram end ``` -## Extension points +## Limitations & Planned Extensions -- **Ontology versioning**: Today `schemaId` is a single UUID; you could version schemas and map multiple local tables to different versions. +- **Ontology versioning**: Today `schemaId` is a single UUID, there are plans for adding a `schemaVersion` key to allow for schema versioning. - **Conflict resolution**: Add version or timestamp fields and resolve conflicts in the adapter or in a separate service. - **Idempotency**: Use idempotency keys in payloads or in the mapping DB to make webhook handling and outbound sync idempotent. -- **ID mapping storage**: Replace SQLite with a shared store if multiple instances of the platform need the same mappings. - **Transactional outbox**: Have the platform write changes to an outbox table and have a worker call `handleChange` so that sync is tied to the same transaction as the local write. diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md index a13f1a7d..a1c9d65c 100644 --- a/docs/docs/W3DS Protocol/Awareness-Protocol.md +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -5,7 +5,7 @@ sidebar_position: 4 # Awareness Protocol :::warning Prototype-level implementation -The Awareness Protocol described here is **prototype-level**. The packet format, delivery behavior, and platform contract are subject to change. Do not rely on them for production without checking for updates. +The Awareness Protocol described here is **prototype-level**. The packet format, delivery behavior, and platform contract are subject to change in upcoming updates ::: The Awareness Protocol is the webhook deliverability mechanism in W3DS. When data in an eVault changes (create or update), the eVault notifies all other registered platforms so they can stay in sync. "Awareness" means platforms become aware of changes that happened elsewhere. @@ -101,11 +101,11 @@ For a step-by-step implementation guide, see the [Webhook Controller Guide](/doc ## Limitations and Extension Points -The current protocol has intentional limitations that readers may improve in their own implementations: +The current protocol has limitations which are going to be improved in subsequent versions: - **No retries**: Failed webhooks are not retried. A more robust system could add retries with backoff or a dead-letter queue. - **No ordering guarantee**: Webhooks to different platforms are sent in parallel; there is no guarantee of order across platforms or across multiple mutations. - **No at-least-once guarantee**: Because delivery is fire-and-forget and there are no retries, a platform might never receive a given update. At-least-once delivery would require acknowledgments and retries (and possibly idempotency keys). -- **Platform list from Registry**: The set of recipients is whatever the Registry returns for `GET /platforms`. How that list is maintained and updated is registry-specific. +- **Platform list from Registry**: The set of recipients is whatever the Registry returns for `GET /platforms`. This is a prototype level shortcut and will be phased out. Designing retries, ordering, or delivery guarantees would be natural extension points for a production-grade awareness mechanism. From 44d8ee8c8a26449c7267e8d0b9b66a8b025b2038 Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Thu, 29 Jan 2026 19:10:02 +0530 Subject: [PATCH 05/10] chore: fix broken link --- docs/docs/Infrastructure/Web3-Adapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md index 8ada7760..fd5f98e3 100644 --- a/docs/docs/Infrastructure/Web3-Adapter.md +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -112,7 +112,7 @@ graph TB ### Mapping configuration (IMapping) -Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the [Web3 Adapter Mapping Rules](../../../infrastructure/web3-adapter/MAPPING_RULES.md). +Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the **Web3 Adapter Mapping Rules** in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. Each mapping is a JSON file with: From 6a967a8fe688ad5e1b75a36cfce522798968574e Mon Sep 17 00:00:00 2001 From: Merul Dhiman <69296233+coodos@users.noreply.github.com> Date: Thu, 29 Jan 2026 19:14:55 +0530 Subject: [PATCH 06/10] Update docs/docs/W3DS Protocol/Awareness-Protocol.md Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- docs/docs/W3DS Protocol/Awareness-Protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md index a1c9d65c..c7a53acf 100644 --- a/docs/docs/W3DS Protocol/Awareness-Protocol.md +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -25,7 +25,7 @@ Platforms do not poll eVaults for changes. Instead, eVault Core pushes change no The Awareness Protocol is triggered after: -1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay**. The delay exists because the requesting platform's URL may not be known immediately from the Bearer token; without the delay, the same platform could receive its own write back and create a feedback loop ("webhook ping-pong"). +1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay** to ensure the requesting platform can be reliably identified and excluded from recipients, preventing the same platform from receiving its own write back and creating a feedback loop ("webhook ping-pong"). 2. **updateMetaEnvelopeById** (update): After the MetaEnvelope is updated, webhooks are sent **immediately** (fire-and-forget, no delay). ## Mechanism From d017a6b704f9d3a2c7c86564daa46d67171f7f0e Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Fri, 30 Jan 2026 12:29:01 +0530 Subject: [PATCH 07/10] docs: fix comments on web3adapter docs --- docs/docs/Infrastructure/Web3-Adapter.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/docs/Infrastructure/Web3-Adapter.md b/docs/docs/Infrastructure/Web3-Adapter.md index fd5f98e3..9bf87be8 100644 --- a/docs/docs/Infrastructure/Web3-Adapter.md +++ b/docs/docs/Infrastructure/Web3-Adapter.md @@ -6,15 +6,22 @@ sidebar_position: 4 The Web3 Adapter is the bridge between a platform's local database and eVault. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. +Web3 Adapter is only needed when you have a database which you want to +automatically sync with eVaults, if your application is stateless or the application only writes to the eVault directly then you don't need a Web3 Adapter. + ## Overview Platforms keep their own schemas and databases. To participate in W3DS, they need a component that: -- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's eVault (via the user's [eName](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). -- **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. +- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's and/or target eVault (via the user's [eName](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). +- **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. See [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for more details. + +Please Note: That neither inbound or outbound are immediate, or have any transactional guarantees. The Web3 Adapter implements this bridge. Understanding its core ideas helps you design better ontologies, mappings, and consistency strategies. +Please note that the web3 adapter may not come pre-assembled with hooks for all databases. + ### Key Features - **Bidirectional mapping**: Local schema ↔ global ontology via JSON mapping configs. @@ -26,7 +33,7 @@ The Web3 Adapter implements this bridge. Understanding its core ideas helps you ### 1. Universal ontology as the common language -All platforms agree on a small set of global schemas (e.g. User, Post, Group) identified by UUIDs. Each platform maps its local tables to these schemas. The ontology is the contract: if you store data in that shape in an eVault, other platforms can interpret it via their own mappings. Better ontology design (versioning, optional fields, extensibility) leads to easier evolution and fewer breaking changes. +All platforms agree on a small set of global schemas (e.g. User, Post, Group) identified by W3IDs. Each platform maps its local tables to these schemas. The ontology is the contract: if you store data in that shape in an eVault, other platforms can interpret it via their own mappings. Better ontology design (versioning, optional fields, extensibility) leads to easier evolution and fewer breaking changes. ### 2. Per-entity owner eVault (W3ID) From 95a638c3aaf03e776443d784e028c31fa4e4cb09 Mon Sep 17 00:00:00 2001 From: Merul Dhiman Date: Fri, 30 Jan 2026 12:40:34 +0530 Subject: [PATCH 08/10] docs: address comments on w3id and awareness --- docs/docs/W3DS Basics/W3ID.md | 3 +++ docs/docs/W3DS Protocol/Awareness-Protocol.md | 15 ++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/docs/W3DS Basics/W3ID.md b/docs/docs/W3DS Basics/W3ID.md index cc7e5d33..d47b9cb4 100644 --- a/docs/docs/W3DS Basics/W3ID.md +++ b/docs/docs/W3DS Basics/W3ID.md @@ -94,4 +94,7 @@ The W3ID system is implemented in the `w3id` package (TypeScript) and provides: - **ID log manager**: Immutable, signed event logs for key rotation and identity updates. - **JWT signing**: A W3ID with a signer can sign JWTs (e.g. for authentication or key binding certificates). +This package is useful to create W3IDs with keys or make them global, it is +consumed currently by [eID Wallet](/docs/Infrastructure/eID-Wallet) and [Web3 Adapter](/docs/Infrastructure/Web3-Adapter) + For implementation details (builder API, storage backends, logging format), see the `w3id` package in the repository. diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md index a1c9d65c..f8ac09c4 100644 --- a/docs/docs/W3DS Protocol/Awareness-Protocol.md +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -8,7 +8,7 @@ sidebar_position: 4 The Awareness Protocol described here is **prototype-level**. The packet format, delivery behavior, and platform contract are subject to change in upcoming updates ::: -The Awareness Protocol is the webhook deliverability mechanism in W3DS. When data in an eVault changes (create or update), the eVault notifies all other registered platforms so they can stay in sync. "Awareness" means platforms become aware of changes that happened elsewhere. +The Awareness Protocol is the webhook delivery mechanism in W3DS. When data in an eVault changes (create or update), the eVault notifies every registered platforms so they can stay in sync. "Awareness" means platforms become aware of changes that happened elsewhere. ## Overview @@ -63,11 +63,17 @@ The body of each webhook request is JSON with the following fields: | Field | Description | |-------|-------------| -| `id` | Global MetaEnvelope ID (UUID). | +| `id` | MetaEnvelope ID (W3ID). | | `w3id` | Owner eName (eVault owner W3ID). | | `schemaId` | Ontology/schema UUID (identifies the type of entity and which mapping the platform should use). | | `data` | The entity payload in the **global ontology** shape (parsed key-value structure). | -| `evaultPublicKey` | Optional; eVault public key. | + +In the current version of the implementation the entitre payload is sent in +plain text to any registered platform, so all data is sent to every platform and +it's the platform's responsibility to reject any packets it doesn't use, in +future versions of awareness protocol, it will be changed so that platforms can +subscribe to certain ontology changes and they will be provided details of the +updated MetaEnvelope by reference instead of value. **Content-Type**: `application/json` @@ -83,8 +89,7 @@ The body of each webhook request is JSON with the following fields: "mediaUrls": [], "authorId": "@e4d909c2-5d2f-4a7d-9473-b34b6c0f1a5a", "createdAt": "2025-01-24T10:00:00Z" - }, - "evaultPublicKey": "z..." + } } ``` From b2ba55a4572fc0f8ab817d2ee54685bbd1193fbf Mon Sep 17 00:00:00 2001 From: Merul Dhiman <69296233+coodos@users.noreply.github.com> Date: Fri, 30 Jan 2026 14:41:50 +0530 Subject: [PATCH 09/10] Update docs/docs/W3DS Protocol/Awareness-Protocol.md Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- docs/docs/W3DS Protocol/Awareness-Protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md index 20d81885..9e789647 100644 --- a/docs/docs/W3DS Protocol/Awareness-Protocol.md +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -68,7 +68,7 @@ The body of each webhook request is JSON with the following fields: | `schemaId` | Ontology/schema UUID (identifies the type of entity and which mapping the platform should use). | | `data` | The entity payload in the **global ontology** shape (parsed key-value structure). | -In the current version of the implementation the entitre payload is sent in +In the current version of the implementation the entire payload is sent in plain text to any registered platform, so all data is sent to every platform and it's the platform's responsibility to reject any packets it doesn't use, in future versions of awareness protocol, it will be changed so that platforms can From ec0d05a97311a7dbedf5ea89e535dff31a6334cc Mon Sep 17 00:00:00 2001 From: Merul Dhiman <69296233+coodos@users.noreply.github.com> Date: Fri, 30 Jan 2026 14:44:54 +0530 Subject: [PATCH 10/10] Update docs/docs/W3DS Protocol/Awareness-Protocol.md Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- docs/docs/W3DS Protocol/Awareness-Protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/W3DS Protocol/Awareness-Protocol.md b/docs/docs/W3DS Protocol/Awareness-Protocol.md index 9e789647..0a94a985 100644 --- a/docs/docs/W3DS Protocol/Awareness-Protocol.md +++ b/docs/docs/W3DS Protocol/Awareness-Protocol.md @@ -25,7 +25,7 @@ Platforms do not poll eVaults for changes. Instead, eVault Core pushes change no The Awareness Protocol is triggered after: -1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay** to ensure the requesting platform can be reliably identified and excluded from recipients, preventing the same platform from receiving its own write back and creating a feedback loop ("webhook ping-pong"). +1. **storeMetaEnvelope** (create): After the new MetaEnvelope is stored in the eVault, webhooks are **scheduled with a 3-second delay** to ensure the requesting platform can be reliably identified and excluded from recipients, preventing the same platform from receiving its own write-back and creating a feedback loop ("webhook ping-pong"). 2. **updateMetaEnvelopeById** (update): After the MetaEnvelope is updated, webhooks are sent **immediately** (fire-and-forget, no delay). ## Mechanism