From 5561b7842bf3bb0fbfe9b6a6796823802fe023b8 Mon Sep 17 00:00:00 2001 From: swarnabhasinha Date: Mon, 1 Dec 2025 16:17:45 +0530 Subject: [PATCH] docs: update guides to match dev setup guide --- docs/guides/01-getting-started-cli.md | 2 +- docs/guides/02-getting-started-docker.md | 44 +++++++++++++++--------- docs/guides/03-parameters.md | 28 +++++++-------- docs/guides/04-experiments.md | 30 ++++++++-------- docs/guides/05-faq-glossary.md | 35 ++++++++++++------- 5 files changed, 79 insertions(+), 60 deletions(-) diff --git a/docs/guides/01-getting-started-cli.md b/docs/guides/01-getting-started-cli.md index 873f6dc..3ea8b35 100644 --- a/docs/guides/01-getting-started-cli.md +++ b/docs/guides/01-getting-started-cli.md @@ -1,6 +1,6 @@ # Getting Started with mump2p CLI -The `mump2p` CLI is the quickest way to interact with [Optimum Network](https://github.com/getoptimum/optimum-p2p) without running your own infrastructure. +The `mump2p` CLI is the quickest way to interact with Optimum Network without running your own infrastructure. In the next 5 minutes, you'll have: diff --git a/docs/guides/02-getting-started-docker.md b/docs/guides/02-getting-started-docker.md index 5773e08..4b308ad 100644 --- a/docs/guides/02-getting-started-docker.md +++ b/docs/guides/02-getting-started-docker.md @@ -62,8 +62,8 @@ Choose the deployment mode that best fits your use case: | Component | Purpose | Docker Images | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | -| **mump2p Node** | RLNC-enabled mesh peer, encodes/decodes message shards, handles peer discovery and subscriptions. Optional gRPC API for direct clients. | `getoptimum/p2pnode:v0.0.1-rc2` | -| **Optimum Proxy** | Bridges clients and the mesh, manages subscriptions, shard reassembly, threshold logic, and node selection. | `getoptimum/proxy:v0.0.1-rc3` | +| **mump2p Node** | RLNC-enabled mesh peer, encodes/decodes message shards, handles peer discovery and subscriptions. Optional gRPC API for direct clients. | `getoptimum/p2pnode:${P2P_NODE_VERSION-latest}` | +| **Optimum Proxy** | Bridges clients and the mesh, manages subscriptions, shard reassembly, threshold logic, and node selection. | `getoptimum/proxy:${PROXY_VERSION-latest}` | @@ -96,13 +96,21 @@ Before starting, create your `.env` file: cp .env.example .env ``` +**Important:** After copying, you need to replace the `BOOTSTRAP_PEER_ID` in your `.env` file with the peer ID generated by `make generate-identity`. + +**Workflow:** + +1. Run `make generate-identity` - this creates a unique peer ID +2. Copy the generated peer ID from the output +3. Edit your `.env` file and replace the example `BOOTSTRAP_PEER_ID` with your generated one + Edit with your values: ```bash BOOTSTRAP_PEER_ID= CLUSTER_ID=my-cluster -PROXY_VERSION=v0.0.1-rc7 -P2P_NODE_VERSION=v0.0.1-rc6 +PROXY_VERSION=v0.0.1-rc16 +P2P_NODE_VERSION=v0.0.1-rc16 ``` > **Complete Guide:** [Environment configuration](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#environment-variables-env) @@ -117,7 +125,7 @@ make generate-identity This creates `./identity/p2p.key` with your unique Peer ID. -> **Complete Guide:** [Identity generation and Makefile commands](https://github.com/getoptimum/optimum-dev-setup-guide#quick-start) - all make commands, direct binary usage +> **Complete Guide:** [Identity generation and Makefile commands](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#getting-started) - all make commands, direct binary usage ## 5. Mode A — OptimumProxy + mump2p (Recommended) @@ -156,20 +164,20 @@ services: > **Complete Docker Compose:** > > * [Full configuration with 2 proxies + 4 nodes](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docker-compose-optimum.yml) -> * [All environment variables](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#p2p-node-variables) +> * [All environment variables](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#environment-variables-env) ### Start the Network ```sh export BOOTSTRAP_PEER_ID= -docker compose up -d +docker-compose -f docker-compose-optimum.yml up --build -d ``` ### Verify Health ```sh # Check containers -docker compose ps +docker-compose -f docker-compose-optimum.yml ps # Test endpoints curl http://localhost:8081/api/v1/health # Proxy @@ -251,12 +259,14 @@ services: ``` > **Complete Docker Compose:** [Full configuration for direct P2P mode](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docker-compose-optimum.yml) +> +> **Note:** The docker-compose file uses environment variables from `.env` for versions. Ensure your `.env` file has `PROXY_VERSION` and `P2P_NODE_VERSION` set. ### Start and Verify ```sh export BOOTSTRAP_PEER_ID= -docker compose up -d +docker-compose -f docker-compose-optimum.yml up --build -d curl http://localhost:9091/api/v1/health ``` @@ -266,15 +276,15 @@ Connect using the P2P client with trace handling and metrics: ```bash # Subscribe -./p2p-client -mode=subscribe -topic=testtopic --addr=127.0.0.1:33221 +./grpc_p2p_client/p2p-client -mode=subscribe -topic=testtopic --addr=127.0.0.1:33221 # Publish -./p2p-client -mode=publish -topic=testtopic -msg="Hello" --addr=127.0.0.1:33222 +./grpc_p2p_client/p2p-client -mode=publish -topic=testtopic -msg="Hello" --addr=127.0.0.1:33222 ``` > **Complete Implementation:** > -> * [P2P Client Source](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/p2p_client.go) +> * [P2P Client Source](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/cmd/single/main.go) > * [Usage Guide with Makefile commands](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#using-p2p-nodes-directly-optional--no-proxy) > * [Message format explanation](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#understanding-message-output-format) @@ -284,8 +294,8 @@ For all configuration variables, see the [Parameters Section](./03-parameters.md ### "Connection refused" from client -* Ensure you’re pointing to the host-mapped ports (e.g., 33221, 8081). -* Run docker compose ps and confirm port bindings. +* Ensure you're pointing to the host-mapped ports (e.g., 33221, 8081). +* Run `docker-compose -f docker-compose-optimum.yml ps` and confirm port bindings. * Firewalls: allow inbound localhost traffic. ### Proxy can’t reach nodes @@ -293,7 +303,7 @@ For all configuration variables, see the [Parameters Section](./03-parameters.md * Inside the proxy container, resolve and ping node hosts: ```sh -docker compose exec proxy sh -lc 'nc -zv p2pnode-1 33212' +docker-compose -f docker-compose-optimum.yml exec proxy-1 sh -lc 'nc -zv p2pnode-1 33212' ``` * Make sure `P2P_NODES` hostnames match the `service names` in compose. @@ -311,11 +321,11 @@ docker compose exec proxy sh -lc 'nc -zv p2pnode-1 33212' Stop: ```sh -docker compose down +docker-compose -f docker-compose-optimum.yml down ``` Full reset (containers, volumes, images created by this compose file): ```sh -docker compose down -v --rmi local +docker-compose -f docker-compose-optimum.yml down -v --rmi local ``` diff --git a/docs/guides/03-parameters.md b/docs/guides/03-parameters.md index 7fe1fda..e1b2098 100644 --- a/docs/guides/03-parameters.md +++ b/docs/guides/03-parameters.md @@ -111,9 +111,9 @@ Example Docker service: proxy-1: image: 'getoptimum/proxy:${PROXY_VERSION-latest}' environment: - - PROXY_HTTP_PORT=:8080 + - PROXY_PORT=:8080 - PROXY_GRPC_PORT=:50051 - - CLUSTER_ID=proxy-1 + - CLUSTER_ID=${CLUSTER_ID} - ENABLE_AUTH=false - LOG_LEVEL=debug - P2P_NODES=p2pnode-1:33212,p2pnode-2:33212 @@ -121,7 +121,7 @@ proxy-1: | Parameter | Default | Purpose | | --------------------------------- | ------- | ---------------------------------------------------------- | -| `PROXY_HTTP_PORT` | :8080 | HTTP API port for clients. | +| `PROXY_PORT` | :8080 | HTTP API port for clients. | | `PROXY_GRPC_PORT` | :50051 | gRPC API port for clients. | | `ENABLE_AUTH` | false | Enable Auth0 authentication. | | `AUTH0_DOMAIN` / `AUTH0_AUDIENCE` | (none) | Auth0 settings (required if `ENABLE_AUTH`=true). | @@ -139,16 +139,15 @@ The following table shows the **production defaults**, which are optimized for t ```yaml environment: - NODE_MODE=optimum - - LOG_LEVEL=production + - LOG_LEVEL=debug - CLUSTER_ID=my-cluster - SIDECAR_PORT=33212 - - API_PORT=8081 + - API_PORT=9090 - IDENTITY_DIR=/identity - OPTIMUM_PORT=7070 - OPTIMUM_MAX_MSG_SIZE=1048576 - - OPTIMUM_RANDOM_MSG_SIZE=512 - OPTIMUM_MESH_TARGET=6 - - OPTIMUM_MESH_MIN=4 + - OPTIMUM_MESH_MIN=3 - OPTIMUM_MESH_MAX=12 - OPTIMUM_SHARD_FACTOR=4 - OPTIMUM_SHARD_MULT=1.5 @@ -161,10 +160,10 @@ environment: ```yaml environment: - NODE_MODE=gossipsub - - LOG_LEVEL=production + - LOG_LEVEL=debug - CLUSTER_ID=my-cluster - SIDECAR_PORT=33212 - - API_PORT=8081 + - API_PORT=9090 - IDENTITY_DIR=/identity - GOSSIPSUB_PORT=6060 - GOSSIPSUB_MAX_MSG_SIZE=1048576 @@ -178,17 +177,16 @@ environment: | Parameter | Default Value | Used In Mode | Description | | ---------------------------- | ------------- | ------------ | ---------------------------------------------- | -| `LOG_LEVEL` | production | Both | Log verbosity (production/debug/info/warn/error) | +| `LOG_LEVEL` | debug | Both | Log verbosity (debug/info/warn/error) | | `CLUSTER_ID` | "" | Both | Logical group name for metrics | | `NODE_MODE` | "" | Both | Protocol mode (optimum/gossipsub) | | `SIDECAR_PORT` | 33212 | Both | gRPC sidecar port | -| `API_PORT` | 8081 | Both | HTTP API port | -| `IDENTITY_DIR` | /tmp | Both | Directory for node private key | +| `API_PORT` | 9090 | Both | HTTP API port | +| `IDENTITY_DIR` | /identity | Both | Directory for node private key | | `OPTIMUM_PORT` | 7070 | optimum | TCP port for RLNC gossip | | `OPTIMUM_MAX_MSG_SIZE` | 1048576 | optimum | Max message size (1MB) | -| `OPTIMUM_RANDOM_MSG_SIZE` | 512 | optimum | Random message size for testing | | `OPTIMUM_MESH_TARGET` | 6 | optimum | Desired peers in mesh | -| `OPTIMUM_MESH_MIN` | 4 | optimum | Minimum peers before adding more | +| `OPTIMUM_MESH_MIN` | 3 | optimum | Minimum peers before adding more | | `OPTIMUM_MESH_MAX` | 12 | optimum | Max peers before pruning | | `OPTIMUM_SHARD_FACTOR` | 4 | optimum | Number of shards per message | | `OPTIMUM_SHARD_MULT` | 1.5 | optimum | Redundancy multiplier (extra shards) | @@ -198,7 +196,7 @@ environment: | `GOSSIPSUB_MESH_TARGET` | 6 | gossipsub | Desired peers in mesh | | `GOSSIPSUB_MESH_MIN` | 4 | gossipsub | Minimum peers before adding more | | `GOSSIPSUB_MESH_MAX` | 12 | gossipsub | Max peers before pruning | -| `BOOTSTRAP_PEERS` | [] | Both | List of peer multiaddrs for bootstrap | +| `BOOTSTRAP_PEERS` | (none) | Both | List of peer multiaddrs for bootstrap | > **Note:** These are the production defaults used by mump2p nodes. For experimental tuning, see [Common Experiments](./04-experiments.md). diff --git a/docs/guides/04-experiments.md b/docs/guides/04-experiments.md index 4adab61..f43a9a5 100644 --- a/docs/guides/04-experiments.md +++ b/docs/guides/04-experiments.md @@ -13,7 +13,7 @@ Each experiment below lists the **goal**, a quick **how-to**, and **what to obse You can run them using: * **mump2p CLI** (see [CLI Guide](./01-getting-started-cli.md)) -* **gRPC client** (with `MessageTraceGossipSub` or `MessageTraceOptimump2p` for protocol metrics) +* **gRPC P2P client** (trace collection is automatic when subscribing - see [Trace Collection](#metrics-collection)) @@ -112,10 +112,7 @@ You can run them using: **Expected Result:** mump2p should handle higher stress levels before failing compared to GossipSub. -> **Tip:** Enable protocol traces in the gRPC client to get hop-by-hop delivery info: -> -> * `MessageTraceGossipSub` for GossipSub mode. -> * `MessageTraceOptimump2p` for mump2p mode. +> **Tip:** Trace collection is automatic when using the gRPC P2P client. Simply subscribe to a topic and trace events will be automatically parsed and displayed. See [Trace Collection](#metrics-collection) for details. ## Metrics Collection @@ -123,22 +120,27 @@ For comprehensive metrics collection during experiments, use the gRPC P2P client **[P2P Client with Metrics Collection](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#collecting-trace-data-for-experiments)** -**[Complete Code](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/p2p_client.go)** +**[Complete Code](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/cmd/single/main.go)** -The client automatically captures and displays: +The client automatically captures and displays trace events when you subscribe: * **GossipSub traces**: Peer routing, message delivery status, hop counts, latency * **mump2p traces**: Shard encoding/decoding, reconstruction efficiency, redundancy metrics * **Message-level data**: Delivery success rates, end-to-end latency, bandwidth usage -**Key trace handlers:** +**Usage:** + +```bash +# Subscribe to a topic - trace events are automatically collected and displayed +./grpc_p2p_client/p2p-client -mode=subscribe -topic=your-topic --addr=127.0.0.1:33221 +``` + +Trace parsing is implemented in `grpc_p2p_client/shared/utils.go` and handles both `ResponseType_MessageTraceGossipSub` and `ResponseType_MessageTraceMumP2P` automatically. -```go -case protobuf.ResponseType_MessageTraceGossipSub: - fmt.Printf("[TRACE] GossipSub trace received: %s\n", string(resp.GetData())) +For multi-node experiments with trace data collection, use the multi-subscribe client: -case protobuf.ResponseType_MessageTraceOptimump2p: - fmt.Printf("[TRACE] mump2p trace received: %s\n", string(resp.GetData())) +```bash +./grpc_p2p_client/p2p-multi-subscribe -topic=test-topic -ipfile=ips.txt -output-data=data.tsv -output-trace=trace.tsv ``` -Use this client instead of the CLI for detailed performance analysis during experiments. +See the [dev setup guide](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#multi-node-client-tools) for more details on multi-node client tools. diff --git a/docs/guides/05-faq-glossary.md b/docs/guides/05-faq-glossary.md index c086b98..06814b7 100644 --- a/docs/guides/05-faq-glossary.md +++ b/docs/guides/05-faq-glossary.md @@ -20,13 +20,17 @@ It includes detailed troubleshooting for: ### Q: `docker run ... generate-key` command doesn't work -**A:** Use the identity generation script instead: +**A:** Use the identity generation script or Makefile command: ```bash -curl -sSL https://raw.githubusercontent.com/getoptimum/optimum-dev-setup-guide/main/script/generate-identity.sh | bash +# Using Makefile (recommended) +make generate-identity + +# Or using the script directly +./script/generate-identity.sh ``` -This generates `p2p.key` and exports `BOOTSTRAP_PEER_ID` automatically. +This generates `./identity/p2p.key` and displays the `BOOTSTRAP_PEER_ID` that you need to add to your `.env` file. ### Container Startup Issues @@ -34,10 +38,10 @@ This generates `p2p.key` and exports `BOOTSTRAP_PEER_ID` automatically. **A:** Common fixes: -1. **Check Docker images**: Use correct versions (`getoptimum/proxy:v0.0.1-rc3`, `getoptimum/p2pnode:v0.0.1-rc2`) +1. **Check Docker images**: Use correct versions from your `.env` file (`PROXY_VERSION=v0.0.1-rc16`, `P2P_NODE_VERSION=v0.0.1-rc16` by default) 2. **Network conflicts**: Change subnet in docker-compose if `172.28.0.0/16` conflicts -3. **Port conflicts**: Ensure ports 8080, 8081, 33221, 9091, 7071 are available -4. **Platform issues**: Add `platform: linux/amd64` for M1 Macs +3. **Port conflicts**: Ensure ports 8081, 8082, 50051, 50052, 33221-33224, 9091-9094, 7071-7074 are available +4. **Platform issues**: Add `platform: linux/amd64` for M1 Macs (already included in docker-compose-optimum.yml) ### Q: "Connection refused" when clients try to connect @@ -60,6 +64,8 @@ This generates `p2p.key` and exports `BOOTSTRAP_PEER_ID` automatically. 3. **Use latest client examples**: Reference [`optimum-dev-setup-guide/docs/guide.md#grpc-proxy-client-implementation`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#grpc-proxy-client-implementation) **[Complete Code](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_proxy_client/proxy_client.go)** + + For P2P direct client, see [`grpc_p2p_client/cmd/single/main.go`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/cmd/single/main.go) ### Q: Getting "method not found" or protobuf errors @@ -67,7 +73,7 @@ This generates `p2p.key` and exports `BOOTSTRAP_PEER_ID` automatically. * See the [API Reference section](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md#api-reference) for complete protobuf definitions * All proto files are available in the repository's `grpc_*_client/proto/` directories: - * [`grpc_proxy_client/proto/gateway_stream.proto`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_proxy_client/proto/gateway_stream.proto) + * [`grpc_proxy_client/proto/proxy_stream.proto`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_proxy_client/proto/proxy_stream.proto) * [`grpc_p2p_client/proto/p2p_stream.proto`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/grpc_p2p_client/proto/p2p_stream.proto) @@ -97,14 +103,17 @@ This generates `p2p.key` and exports `BOOTSTRAP_PEER_ID` automatically. When something doesn't work: -1. **Check container logs**: `docker logs ` -2. **Verify network connectivity**: `docker network ls` and `docker network inspect` -3. **Test basic connectivity**: `curl http://localhost:8080/health` -4. **Check authentication**: `mump2p whoami` -5. **Verify versions**: Use latest CLI and Docker images +1. **Check container logs**: `docker-compose -f docker-compose-optimum.yml logs ` or `docker logs ` +2. **Verify network connectivity**: `docker network ls` and `docker network inspect optimum-dev-setup-guide_optimum-network` +3. **Test basic connectivity**: + * Proxy: `curl http://localhost:8081/api/v1/health` + * P2P Node: `curl http://localhost:9091/api/v1/health` +4. **Check authentication**: `mump2p whoami` (if using CLI) +5. **Verify versions**: Check your `.env` file for `PROXY_VERSION` and `P2P_NODE_VERSION` (default: v0.0.1-rc16) +6. **Check service status**: `docker-compose -f docker-compose-optimum.yml ps` ### Getting Help * **CLI Issues**: [mump2p-cli FAQ](https://github.com/getoptimum/mump2p-cli#faq---common-issues--troubleshooting) * **Setup Issues**: Check [`optimum-dev-setup-guide/docs/guide.md`](https://github.com/getoptimum/optimum-dev-setup-guide/blob/main/docs/guide.md) -* **Protocol Questions**: See [mump2p Documentation](./p2p.md) +* **Protocol Questions**: See [mump2p Documentation](../learn/overview/p2p.md)