diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 00000000..d53fbeb2 --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,223 @@ +# Install OpenViking + +Quick installation guide for OpenViking - the Context Database for AI Agents. + +**Goal:** Get OpenViking running with skills in 5 minutes. + +--- + +## Quick Install + +### 1. Install uv + +**macOS/Linux:** +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +**Windows:** +```powershell +powershell -c "irm https://astral.sh/uv/install.ps1 | iex" +``` + +### 2. Install OpenViking Server + +```bash +uv tool install openviking +``` + +This installs `openviking-server` as a standalone tool you can run anywhere. + +### 3. Install ov CLI (Required for Skills) + +```bash +curl -fsSL https://raw.githubusercontent.com/volcengine/OpenViking/main/crates/ov_cli/install.sh | bash +``` + +Or install from source: +```bash +git clone https://github.com/volcengine/OpenViking.git +cd OpenViking +cargo install --path crates/ov_cli +``` + +### 4. Configure and Start Server + +Create config directory and config file: + +```bash +mkdir -p ~/.openviking + +cat > ~/.openviking/ov.conf << 'EOF' +{ + "embedding": { + "dense": { + "provider": "volcengine", + "model": "doubao-embedding-vision-250615", + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3", + "dimension": 1024, + "input": "multimodal" + } + }, + "vlm": { + "provider": "volcengine", + "model": "doubao-seed-1-8-251228", + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3" + }, + "storage": { + "vectordb": { "backend": "local" }, + "agfs": { "backend": "local" } + } +} +EOF +``` + +**Replace `your-api-key` with your actual API key.** + +> 💡 **Note:** Data is stored in `./data` by default. You can change this by setting the `OPENVIKING_CONFIG_DIR` environment variable before starting the server. + +Start the server in background: + +```bash +export OPENVIKING_CONFIG_DIR=~/.openviking +nohup openviking-server > ~/.openviking/server.log 2>&1 & +``` + +Check it's running: +```bash +curl http://localhost:1933/health +# Should return: {"status": "ok"} +``` + +### 5. Configure CLI + +```bash +cat > ~/.openviking/ovcli.conf << 'EOF' +{ + "url": "http://localhost:1933" +} +EOF +``` + +### 6. Install Skills + +Skills enable agents to use OpenViking. Install them to your agent's skill directory: + +> 🤖 **For AI Agents reading this:** Ask your user if they want to install the OpenViking skills (adding-memory, adding-resource, searching-context). Proceed only if they confirm. + +```bash +mkdir -p ~/.openclaw/skills + +# Copy skills from OpenViking examples +cp -r /path/to/openviking/examples/skills/adding-memory ~/.openclaw/skills/ +cp -r /path/to/openviking/examples/skills/adding-resource ~/.openclaw/skills/ +cp -r /path/to/openviking/examples/skills/searching-context ~/.openclaw/skills/ +``` + +Or download directly: +```bash +mkdir -p ~/.openclaw/skills/adding-memory +curl -o ~/.openclaw/skills/adding-memory/SKILL.md \ + https://raw.githubusercontent.com/volcengine/OpenViking/main/examples/skills/adding-memory/SKILL.md + +mkdir -p ~/.openclaw/skills/adding-resource +curl -o ~/.openclaw/skills/adding-resource/SKILL.md \ + https://raw.githubusercontent.com/volcengine/OpenViking/main/examples/skills/adding-resource/SKILL.md + +mkdir -p ~/.openclaw/skills/searching-context +curl -o ~/.openclaw/skills/searching-context/SKILL.md \ + https://raw.githubusercontent.com/volcengine/OpenViking/main/examples/skills/searching-context/SKILL.md +``` + +--- + +## Using OpenViking Memory + +Once skills are installed, you can use natural language to trigger OpenViking actions: + +### Storing Memories +Say things like: +- "**Remember this**" — after sharing something worth remembering +- "**Save this to memory**" — to persist an insight or decision +- "**Keep this in mind**" — to store context for future reference + +### Adding Resources +Say things like: +- "**Add this to OpenViking**" — when sharing a URL or file +- "**Import https://example.com/docs**" — to add external knowledge +- "**Save this resource**" — to store documents for later retrieval + +### Searching Context +Say things like: +- "**Search my memory for...**" — to find previously stored information +- "**What do I know about...**" — to query your OpenViking context +- "**Find in OpenViking...**" — to search across memories and resources + +The agent will automatically detect these intents and use the appropriate OpenViking skills. + +--- + +## Quick Test + +Verify everything works: + +```bash +# Test CLI connection +ov system health + +# Test adding memory +ov add-memory "Test: OpenViking is working" + +# Test searching +ov search "OpenViking working" +``` + +--- + +## Advanced Configuration + +For advanced setup options (cloud deployment, custom storage, multiple model providers, etc.), see: + +**[INSTALL_ADVANCED.md](./INSTALL_ADVANCED.md)** + +This includes: +- Full configuration reference +- Cloud deployment guides +- Docker/container setup +- Multiple model providers +- Authentication and security +- Troubleshooting deep dives + +--- + +## Requirements + +- Python 3.10+ +- API keys for VLM and embedding models + +**Supported Model Providers:** Volcengine, OpenAI, Anthropic, DeepSeek, Google, Moonshot, Zhipu, DashScope, MiniMax, OpenRouter, vLLM + +--- + +## Quick Reference + +```bash +# Install +uv tool install openviking +curl -fsSL .../install.sh | bash # ov CLI + +# Start server (background) +export OPENVIKING_CONFIG_DIR=~/.openviking +nohup openviking-server > ~/.openviking/server.log 2>&1 & + +# Stop server +pkill openviking-server + +# CLI commands +ov system health # Check server +ov add-memory "text" # Add memory +ov add-resource # Add resource +ov search "query" # Search context +``` diff --git a/INSTALL_ADVANCED.md b/INSTALL_ADVANCED.md new file mode 100644 index 00000000..bd8c86b3 --- /dev/null +++ b/INSTALL_ADVANCED.md @@ -0,0 +1,688 @@ +# Advanced Installation & Configuration + +Detailed configuration and deployment options for OpenViking. + +For the quick start guide, see **[INSTALL.md](./INSTALL.md)**. + +--- + +## Table of Contents + +- [Full Configuration Reference](#full-configuration-reference) +- [Alternative Installation Methods](#alternative-installation-methods) +- [Building from Source](#building-from-source) +- [Cloud Deployment](#cloud-deployment) +- [Docker/Container Setup](#dockercontainer-setup) +- [Multiple Model Providers](#multiple-model-providers) +- [Authentication and Security](#authentication-and-security) +- [Advanced Troubleshooting](#advanced-troubleshooting) + +--- + +## Full Configuration Reference + +### Complete ov.conf Template + +```json +{ + "server": { + "host": "0.0.0.0", + "port": 1933, + "api_key": null, + "cors_origins": ["*"] + }, + "storage": { + "workspace": "./data", + "vectordb": { + "name": "context", + "backend": "local", + "project": "default", + "volcengine": { + "region": "cn-beijing", + "ak": null, + "sk": null + } + }, + "agfs": { + "port": 1833, + "log_level": "warn", + "backend": "local", + "timeout": 10, + "retry_times": 3, + "s3": { + "bucket": null, + "region": null, + "access_key": null, + "secret_key": null, + "endpoint": null, + "prefix": "", + "use_ssl": true + } + } + }, + "embedding": { + "dense": { + "provider": "volcengine", + "model": "doubao-embedding-vision-250615", + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3", + "dimension": 1024, + "input": "multimodal" + } + }, + "vlm": { + "provider": "volcengine", + "model": "doubao-seed-1-8-251228", + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3", + "temperature": 0.0, + "max_retries": 2, + "thinking": false + }, + "rerank": { + "ak": null, + "sk": null, + "host": "api-vikingdb.vikingdb.cn-beijing.volces.com", + "model_name": "doubao-seed-rerank", + "model_version": "251028", + "threshold": 0.1 + }, + "auto_generate_l0": true, + "auto_generate_l1": true, + "default_search_mode": "thinking", + "default_search_limit": 3, + "enable_memory_decay": true, + "memory_decay_check_interval": 3600, + "log": { + "level": "INFO", + "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s", + "output": "stdout", + "rotation": true, + "rotation_days": 3, + "rotation_interval": "midnight" + }, + "parsers": { + "pdf": { + "strategy": "auto", + "max_content_length": 100000, + "max_section_size": 4000, + "section_size_flexibility": 0.3, + "mineru_endpoint": "https://mineru.example.com/api/v1", + "mineru_api_key": "{your-mineru-api-key}", + "mineru_timeout": 300.0 + }, + "code": { + "enable_ast": true, + "extract_functions": true, + "extract_classes": true, + "extract_imports": true, + "include_comments": true, + "max_line_length": 1000, + "max_token_limit": 50000, + "truncation_strategy": "head", + "warn_on_truncation": true + }, + "image": { + "enable_ocr": false, + "enable_vlm": true, + "ocr_lang": "eng", + "vlm_model": "gpt-4-vision", + "max_dimension": 2048 + }, + "audio": { + "enable_transcription": true, + "transcription_model": "whisper-large-v3", + "language": null, + "extract_metadata": true + }, + "video": { + "extract_frames": true, + "frame_interval": 10.0, + "enable_transcription": true, + "enable_vlm_description": false, + "max_duration": 3600.0 + }, + "markdown": { + "preserve_links": true, + "extract_frontmatter": true, + "include_metadata": true, + "max_heading_depth": 3 + }, + "html": { + "extract_text_only": false, + "preserve_structure": true, + "clean_html": true, + "extract_metadata": true + }, + "text": { + "detect_language": true, + "split_by_paragraphs": true, + "max_paragraph_length": 1000, + "preserve_line_breaks": false + } + } +} +``` + +### Configuration Fields + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `server.host` | string | `"0.0.0.0"` | Server bind address | +| `server.port` | int | `1933` | Server port | +| `server.api_key` | string | `null` | API key for authentication | +| `storage.workspace` | string | `"./data"` | Main data directory | +| `storage.vectordb.backend` | string | `"local"` | Vector DB backend: local, volcengine | +| `storage.agfs.backend` | string | `"local"` | AGFS backend: local, s3 | +| `embedding.dense.provider` | string | `"volcengine"` | Embedding provider | +| `embedding.dense.dimension` | int | `1024` | Vector dimension | +| `vlm.provider` | string | `"volcengine"` | VLM provider | +| `vlm.temperature` | float | `0.0` | Model temperature | +| `auto_generate_l0` | bool | `true` | Auto-generate abstracts | +| `auto_generate_l1` | bool | `true` | Auto-generate overviews | +| `log.level` | string | `"INFO"` | Log level: DEBUG, INFO, WARN, ERROR | + +### Environment Variables + +```bash +# Set config file location +export OPENVIKING_CONFIG_FILE=~/.openviking/ov.conf + +# Or set config directory (looks for ov.conf in that dir) +export OPENVIKING_CONFIG_DIR=~/.openviking + +# Set CLI config location +export OPENVIKING_CLI_CONFIG_FILE=~/.openviking/ovcli.conf +``` + +--- + +## Alternative Installation Methods + +### Using pip + +While uv is the recommended method, you can also use pip: + +```bash +pip install openviking +``` + +### Development Installation + +Install from source for development: + +```bash +git clone https://github.com/volcengine/OpenViking.git +cd OpenViking +uv pip install -e ".[dev,test]" +``` + +### Virtual Environment (manual) + +If not using uv's automatic virtualenv: + +```bash +python -m venv venv +source venv/bin/activate # Linux/macOS +# or +venv\Scripts\activate # Windows + +pip install openviking +``` + +--- + +## Building from Source + +> ⚠️ **Note:** Building from source is only needed for development or if you need to modify the code. For regular use, use `uv tool install` or `pip install` to get pre-built wheels. + +### Prerequisites + +- Python 3.10+ +- Rust (for ov CLI) +- Go 1.21+ (for AGFS server) +- CMake 3.15+ +- GCC/G++ or Clang + +### Build Steps + +```bash +git clone https://github.com/volcengine/OpenViking.git +cd OpenViking + +# Install Python package (builds AGFS and C++ extensions) +pip install . + +# Or with uv +uv pip install . + +# Install ov CLI +cargo install --path crates/ov_cli +``` + +### AGFS Build Details + +The AGFS (Agent Filesystem) server is written in Go and is automatically built during installation: + +```bash +# AGFS source location +third_party/agfs/agfs-server/ + +# To build manually: +cd third_party/agfs/agfs-server +go build -o build/agfs-server cmd/server/main.go +``` + +### C++ Extensions + +OpenViking includes C++ extensions for high-performance vector operations: + +```bash +# CMake is used automatically during pip install +# To build manually: +mkdir build && cd build +cmake .. +make -j$(nproc) +``` + +--- + +## Cloud Deployment + +### Volcengine ECS (Recommended) + +See detailed guide: `docs/en/getting-started/03-quickstart-server.md` + +Quick setup: + +```bash +# 1. Create ECS instance with veLinux 2.0 +# 2. Mount data disk to /data +mkdir -p /data + +# 3. Install uv and openviking +curl -LsSf https://astral.sh/uv/install.sh | sh +uv tool install openviking + +# 4. Configure +mkdir -p ~/.openviking +# Create ov.conf with your API keys + +# 5. Start with nohup +export OPENVIKING_CONFIG_DIR=~/.openviking +nohup openviking-server > /data/openviking.log 2>&1 & +``` + +### AWS/GCP/Azure + +General cloud VM setup: + +```bash +# Update system +sudo apt update && sudo apt upgrade -y # Ubuntu/Debian +# or +sudo yum update -y # CentOS/RHEL + +# Install dependencies +sudo apt install -y curl git # Ubuntu/Debian + +# Install uv +curl -LsSf https://astral.sh/uv/install.sh | sh +source $HOME/.cargo/env + +# Install openviking +uv tool install openviking + +# Configure firewall (example for UFW) +sudo ufw allow 1933/tcp + +# Setup systemd service (see below) +``` + +### Systemd Service + +Create `/etc/systemd/system/openviking.service`: + +```ini +[Unit] +Description=OpenViking Server +After=network.target + +[Service] +Type=simple +User=openviking +Environment=OPENVIKING_CONFIG_DIR=/home/openviking/.openviking +ExecStart=/home/openviking/.cargo/bin/openviking-server +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +Enable and start: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable openviking +sudo systemctl start openviking +sudo systemctl status openviking +``` + +--- + +## Docker/Container Setup + +### Docker Run + +```bash +docker run -d \ + --name openviking \ + -p 1933:1933 \ + -v $(pwd)/data:/data \ + -v $(pwd)/ov.conf:/app/ov.conf \ + -e OPENVIKING_CONFIG_FILE=/app/ov.conf \ + volcengine/openviking:latest +``` + +### Docker Compose + +```yaml +version: '3.8' + +services: + openviking: + image: volcengine/openviking:latest + ports: + - "1933:1933" + volumes: + - ./data:/data + - ./ov.conf:/app/ov.conf + environment: + - OPENVIKING_CONFIG_FILE=/app/ov.conf + restart: unless-stopped +``` + +### Kubernetes + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: openviking +spec: + replicas: 1 + selector: + matchLabels: + app: openviking + template: + metadata: + labels: + app: openviking + spec: + containers: + - name: openviking + image: volcengine/openviking:latest + ports: + - containerPort: 1933 + env: + - name: OPENVIKING_CONFIG_FILE + value: /app/ov.conf + volumeMounts: + - name: config + mountPath: /app/ov.conf + subPath: ov.conf + - name: data + mountPath: /data + volumes: + - name: config + configMap: + name: openviking-config + - name: data + persistentVolumeClaim: + claimName: openviking-data +--- +apiVersion: v1 +kind: Service +metadata: + name: openviking +spec: + selector: + app: openviking + ports: + - port: 1933 + targetPort: 1933 +``` + +--- + +## Multiple Model Providers + +### OpenAI Configuration + +```json +{ + "embedding": { + "dense": { + "provider": "openai", + "model": "text-embedding-3-large", + "api_key": "sk-...", + "api_base": "https://api.openai.com/v1", + "dimension": 3072 + } + }, + "vlm": { + "provider": "openai", + "model": "gpt-4-vision-preview", + "api_key": "sk-...", + "api_base": "https://api.openai.com/v1" + } +} +``` + +### Anthropic Configuration + +```json +{ + "vlm": { + "provider": "anthropic", + "model": "claude-3-opus-20240229", + "api_key": "sk-ant-...", + "api_base": "https://api.anthropic.com" + } +} +``` + +### Local vLLM + +```bash +# Start vLLM server +vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 +``` + +```json +{ + "vlm": { + "provider": "vllm", + "model": "meta-llama/Llama-3.1-8B-Instruct", + "api_key": "dummy", + "api_base": "http://localhost:8000/v1" + } +} +``` + +--- + +## Authentication and Security + +### Enable API Key Authentication + +```json +{ + "server": { + "host": "0.0.0.0", + "port": 1933, + "api_key": "your-secure-api-key", + "cors_origins": ["https://yourdomain.com"] + } +} +``` + +### CLI with Authentication + +```json +{ + "url": "http://localhost:1933", + "api_key": "your-secure-api-key" +} +``` + +### HTTPS/SSL + +Use a reverse proxy like Nginx or Caddy: + +```nginx +server { + listen 443 ssl; + server_name openviking.yourdomain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + location / { + proxy_pass http://localhost:1933; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + } +} +``` + +--- + +## Advanced Troubleshooting + +### Debug Logging + +```json +{ + "log": { + "level": "DEBUG", + "output": "file", + "file_path": "/data/openviking.log" + } +} +``` + +### Vector DB Issues + +**Reset local vector DB:** + +```bash +# Stop server +pkill openviking-server + +# Remove vector DB data +rm -rf ~/.openviking/data/vectordb + +# Restart server +openviking-server +``` + +### Memory Issues + +For large deployments, adjust: + +```json +{ + "parsers": { + "pdf": { + "max_content_length": 50000, + "max_section_size": 2000 + }, + "code": { + "max_token_limit": 25000 + } + } +} +``` + +### Port Conflicts + +If port 1933 is taken: + +```json +{ + "server": { + "port": 8080 + }, + "storage": { + "agfs": { + "port": 8081 + } + } +} +``` + +### Connection Issues from Remote Clients + +1. Check firewall rules +2. Verify server binds to `0.0.0.0` (not `127.0.0.1`) +3. Test with curl from remote machine: + ```bash + curl http://server-ip:1933/health + ``` + +--- + +## Performance Tuning + +### For High Throughput + +```json +{ + "server": { + "workers": 4 + }, + "storage": { + "vectordb": { + "backend": "volcengine" + } + } +} +``` + +### For Low Memory + +```json +{ + "embedding": { + "dense": { + "dimension": 512 + } + }, + "auto_generate_l1": false +} +``` + +--- + +## Migration Guides + +### From Vector-only RAG + +See migration guide in documentation. + +### From Other Memory Systems + +Use import scripts to migrate existing data: + +```python +import openviking as ov + +client = ov.SyncOpenViking(path="./data") +client.initialize() + +# Import from existing source +client.add_resource(path="/path/to/existing/data") +``` diff --git a/README.md b/README.md index bed8fb71..0f22549c 100644 --- a/README.md +++ b/README.md @@ -53,45 +53,42 @@ With OpenViking, developers can build an Agent's brain just like managing local --- -## Quick Start +## Installation -### Prerequisites +**Quick Start (5 minutes):** -Before starting with OpenViking, please ensure your environment meets the following requirements: +```bash +# Install uv +curl -LsSf https://astral.sh/uv/install.sh | sh -- **Python Version**: 3.10 or higher -- **Operating System**: Linux, macOS, Windows -- **Network Connection**: A stable network connection is required (for downloading dependencies and accessing model services) +# Install OpenViking server +uv tool install openviking -### 1. Installation +# Install ov CLI for skills +curl -fsSL https://raw.githubusercontent.com/volcengine/OpenViking/main/crates/ov_cli/install.sh | bash -#### Python Package +# Configure and start +mkdir -p ~/.openviking +# Create ~/.openviking/ov.conf with your API keys +# See INSTALL.md for full config template -```bash -pip install openviking +export OPENVIKING_CONFIG_DIR=~/.openviking +openviking-server ``` -#### Rust CLI (Optional) - -```bash -curl -fsSL https://raw.githubusercontent.com/volcengine/OpenViking/main/crates/ov_cli/install.sh | bash -``` +📖 **[Full Installation Guide →](./INSTALL.md)** -Or build from source: +For advanced configuration (cloud deployment, Docker, multiple model providers), see **[INSTALL_ADVANCED.md](./INSTALL_ADVANCED.md)**. -```bash -cargo install --git https://github.com/volcengine/OpenViking ov_cli -``` +--- -### 2. Model Preparation +## Model Preparation OpenViking requires the following model capabilities: - **VLM Model**: For image and content understanding - **Embedding Model**: For vectorization and semantic retrieval -#### Supported VLM Providers - -OpenViking supports multiple VLM providers: +### Supported VLM Providers | Provider | Model | Get API Key | |----------|-------|-------------| @@ -109,7 +106,7 @@ OpenViking supports multiple VLM providers: > 💡 **Tip**: OpenViking uses a **Provider Registry** for unified model access. The system automatically detects the provider based on model name keywords, so you can switch between providers seamlessly. -#### Provider-Specific Notes +### Provider-Specific Notes
Volcengine (Doubao) @@ -122,7 +119,7 @@ Volcengine supports both model names and endpoint IDs. Using model names is reco "provider": "volcengine", "model": "doubao-seed-1-6-240615", "api_key": "your-api-key", - "api_base" : "https://ark.cn-beijing.volces.com/api/v3", + "api_base" : "https://ark.cn-beijing.volces.com/api/v3" } } ``` @@ -135,7 +132,7 @@ You can also use endpoint IDs (found in [Volcengine ARK Console](https://console "provider": "volcengine", "model": "ep-20241220174930-xxxxx", "api_key": "your-api-key", - "api_base" : "https://ark.cn-beijing.volces.com/api/v3", + "api_base" : "https://ark.cn-beijing.volces.com/api/v3" } } ``` @@ -201,121 +198,9 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
-### 3. Environment Configuration - -#### Configuration Template - -Create a configuration file `~/.openviking/ov.conf`: - -```json -{ - "embedding": { - "dense": { - "api_base" : "", // API endpoint address - "api_key" : "", // Model service API Key - "provider" : "", // Provider type: "volcengine" or "openai" (currently supported) - "dimension": 1024, // Vector dimension - "model" : "" // Embedding model name (e.g., doubao-embedding-vision-250615 or text-embedding-3-large) - } - }, - "vlm": { - "api_base" : "", // API endpoint address - "api_key" : "", // Model service API Key - "provider" : "", // Provider type (volcengine, openai, deepseek, anthropic, etc.) - "model" : "" // VLM model name (e.g., doubao-seed-1-8-251228 or gpt-4-vision-preview) - } -} -``` - -> **Note**: For embedding models, currently `volcengine` (Doubao), `openai`, and `jina` providers are supported. For VLM models, we support multiple providers including volcengine, openai, deepseek, anthropic, gemini, moonshot, zhipu, dashscope, minimax, and more. - -#### Configuration Examples - -👇 Expand to see the configuration example for your model service: - -
-Example 1: Using Volcengine (Doubao Models) - -```json -{ - "embedding": { - "dense": { - "api_base" : "https://ark.cn-beijing.volces.com/api/v3", - "api_key" : "your-volcengine-api-key", - "provider" : "volcengine", - "dimension": 1024, - "model" : "doubao-embedding-vision-250615" - } - }, - "vlm": { - "api_base" : "https://ark.cn-beijing.volces.com/api/v3", - "api_key" : "your-volcengine-api-key", - "provider" : "volcengine", - "model" : "doubao-seed-1-8-251228" - } -} -``` - -
- -
-Example 2: Using OpenAI Models - -```json -{ - "embedding": { - "dense": { - "api_base" : "https://api.openai.com/v1", - "api_key" : "your-openai-api-key", - "provider" : "openai", - "dimension": 3072, - "model" : "text-embedding-3-large" - } - }, - "vlm": { - "api_base" : "https://api.openai.com/v1", - "api_key" : "your-openai-api-key", - "provider" : "openai", - "model" : "gpt-4-vision-preview" - } -} -``` - -
- -#### Set Environment Variable - -After creating the configuration file, set the environment variable to point to it (Linux/macOS): - -```bash -export OPENVIKING_CONFIG_FILE=~/.openviking/ov.conf -``` - -On Windows, use one of the following: - -PowerShell: - -```powershell -$env:OPENVIKING_CONFIG_FILE = "$HOME/.openviking/ov.conf" -``` - -Command Prompt (cmd.exe): - -```bat -set "OPENVIKING_CONFIG_FILE=%USERPROFILE%\.openviking\ov.conf" -``` - -> 💡 **Tip**: You can also place the configuration file in other locations, just specify the correct path in the environment variable. - -### 4. Run Your First Example - -> 📝 **Prerequisite**: Ensure you have completed the environment configuration in the previous step. - -Now let's run a complete example to experience the core features of OpenViking. - -#### Create Python Script +--- -Create `example.py`: +## Quick Example ```python import openviking as ov @@ -365,225 +250,135 @@ except Exception as e: print(f"Error: {e}") ``` -#### Run the Script - -```bash -python example.py -``` - -#### Expected Output - -``` -Directory structure: -... - -Content preview: ... +--- -Wait for semantic processing... -Abstract: -... +## Skills -Overview: -... +Skills enable agents to use OpenViking through simple keywords: -Search results: - viking://resources/... (score: 0.8523) - ... -``` +| Keyword | Skill | What It Does | +|---------|-------|--------------| +| `ovm` | adding-memory | Extracts and stores insights from conversation | +| `ovr` | adding-resource | Imports files/URLs into OpenViking | +| `ovs` | searching-context | Searches stored memories and resources | -Congratulations! You have successfully run OpenViking 🎉 +See [INSTALL.md](./INSTALL.md) for skill installation instructions. --- ## Server Deployment -For production environments, we recommend running OpenViking as a standalone HTTP service to provide persistent, high-performance context support for your AI Agents. +For production environments, we recommend running OpenViking as a standalone HTTP service: -🚀 **Deploy OpenViking on Cloud**: -To ensure optimal storage performance and data security, we recommend deploying on **Volcengine Elastic Compute Service (ECS)** using the **veLinux** operating system. We have prepared a detailed step-by-step guide to get you started quickly. +```bash +# Start server +export OPENVIKING_CONFIG_DIR=~/.openviking +openviking-server -👉 **[View: Server Deployment & ECS Setup Guide](./docs/en/getting-started/03-quickstart-server.md)** +# Or run in background +nohup openviking-server > ~/.openviking/server.log 2>&1 & +``` + +🚀 For cloud deployment (ECS, Docker, Kubernetes), see **[INSTALL_ADVANCED.md](./INSTALL_ADVANCED.md)**. --- ## Core Concepts -After running the first example, let's dive into the design philosophy of OpenViking. These five core concepts correspond one-to-one with the solutions mentioned earlier, together building a complete context management system: - ### 1. Filesystem Management Paradigm → Solves Fragmentation We no longer view context as flat text slices but unify them into an abstract virtual filesystem. Whether it's memories, resources, or capabilities, they are mapped to virtual directories under the `viking://` protocol, each with a unique URI. -This paradigm gives Agents unprecedented context manipulation capabilities, enabling them to locate, browse, and manipulate information precisely and deterministically through standard commands like `ls` and `find`, just like a developer. This transforms context management from vague semantic matching into intuitive, traceable "file operations". Learn more: [Viking URI](./docs/en/concepts/04-viking-uri.md) | [Context Types](./docs/en/concepts/02-context-types.md) - ``` viking:// -├── resources/ # Resources: project docs, repos, web pages, etc. -│ ├── my_project/ -│ │ ├── docs/ -│ │ │ ├── api/ -│ │ │ └── tutorials/ -│ │ └── src/ -│ └── ... -├── user/ # User: personal preferences, habits, etc. +├── resources/ # Resources: project docs, repos, web pages, etc. +│ └── my_project/ +│ ├── docs/ +│ └── src/ +├── user/ # User: personal preferences, habits, etc. │ └── memories/ -│ ├── preferences/ -│ │ ├── writing_style -│ │ └── coding_habits -│ └── ... -└── agent/ # Agent: skills, instructions, task memories, etc. +│ └── preferences/ +└── agent/ # Agent: skills, instructions, task memories, etc. ├── skills/ - │ ├── search_code - │ ├── analyze_data - │ └── ... - ├── memories/ - └── instructions/ + └── memories/ ``` +Learn more: [Viking URI](./docs/en/concepts/04-viking-uri.md) | [Context Types](./docs/en/concepts/02-context-types.md) + ### 2. Tiered Context Loading → Reduces Token Consumption -Stuffing massive amounts of context into a prompt all at once is not only expensive but also prone to exceeding model windows and introducing noise. OpenViking automatically processes context into three levels upon writing: -- **L0 (Abstract)**: A one-sentence summary for quick retrieval and identification. -- **L1 (Overview)**: Contains core information and usage scenarios for Agent decision-making during the planning phase. -- **L2 (Details)**: The full original data, for deep reading by the Agent when absolutely necessary. +OpenViking automatically processes context into three levels upon writing: -Learn more: [Context Layers](./docs/en/concepts/03-context-layers.md) +- **L0 (Abstract)**: A one-sentence summary for quick retrieval and identification (~100 tokens) +- **L1 (Overview)**: Contains core information and usage scenarios for Agent decision-making (~2k tokens) +- **L2 (Details)**: The full original data, for deep reading when absolutely necessary -``` -viking://resources/my_project/ -├── .abstract # L0 Layer: Abstract (~100 tokens) - Quick relevance check -├── .overview # L1 Layer: Overview (~2k tokens) - Understand structure and key points -├── docs/ -│ ├── .abstract # Each directory has corresponding L0/L1 layers -│ ├── .overview -│ ├── api/ -│ │ ├── .abstract -│ │ ├── .overview -│ │ ├── auth.md # L2 Layer: Full content - Load on demand -│ │ └── endpoints.md -│ └── ... -└── src/ - └── ... -``` +Learn more: [Context Layers](./docs/en/concepts/03-context-layers.md) ### 3. Directory Recursive Retrieval → Improves Retrieval Effect -Single vector retrieval struggles with complex query intents. OpenViking has designed an innovative **Directory Recursive Retrieval Strategy** that deeply integrates multiple retrieval methods: +Single vector retrieval struggles with complex query intents. OpenViking uses an innovative Directory Recursive Retrieval Strategy: -1. **Intent Analysis**: Generate multiple retrieval conditions through intent analysis. -2. **Initial Positioning**: Use vector retrieval to quickly locate the high-score directory where the initial slice is located. -3. **Refined Exploration**: Perform a secondary retrieval within that directory and update high-score results to the candidate set. -4. **Recursive Drill-down**: If subdirectories exist, recursively repeat the secondary retrieval steps layer by layer. -5. **Result Aggregation**: Finally, obtain the most relevant context to return. +1. **Intent Analysis**: Generate multiple retrieval conditions +2. **Initial Positioning**: Use vector retrieval to locate high-score directories +3. **Refined Exploration**: Perform secondary retrieval within directories +4. **Recursive Drill-down**: Repeat for subdirectories +5. **Result Aggregation**: Return the most relevant context -This "lock high-score directory first, then refine content exploration" strategy not only finds the semantically best-matching fragments but also understands the full context where the information resides, thereby improving the globality and accuracy of retrieval. Learn more: [Retrieval Mechanism](./docs/en/concepts/07-retrieval.md) +Learn more: [Retrieval Mechanism](./docs/en/concepts/07-retrieval.md) ### 4. Visualized Retrieval Trajectory → Observable Context -OpenViking's organization uses a hierarchical virtual filesystem structure. All context is integrated in a unified format, and each entry corresponds to a unique URI (like a `viking://` path), breaking the traditional flat black-box management mode with a clear hierarchy that is easy to understand. - -The retrieval process adopts a directory recursive strategy. The trajectory of directory browsing and file positioning for each retrieval is fully preserved, allowing users to clearly observe the root cause of problems and guide the optimization of retrieval logic. Learn more: [Retrieval Mechanism](./docs/en/concepts/07-retrieval.md) +The retrieval trajectory is fully preserved, allowing users to clearly observe the root cause of issues and guide retrieval logic optimization. ### 5. Automatic Session Management → Context Self-Iteration -OpenViking has a built-in memory self-iteration loop. At the end of each session, developers can actively trigger the memory extraction mechanism. The system will asynchronously analyze task execution results and user feedback, and automatically update them to the User and Agent memory directories. +At the end of each session, the system asynchronously analyzes task execution results and automatically updates User and Agent memory directories, making the Agent "smarter with use." -- **User Memory Update**: Update memories related to user preferences, making Agent responses better fit user needs. -- **Agent Experience Accumulation**: Extract core content such as operational tips and tool usage experience from task execution experience, aiding efficient decision-making in subsequent tasks. - -This allows the Agent to get "smarter with use" through interactions with the world, achieving self-evolution. Learn more: [Session Management](./docs/en/concepts/08-session.md) +Learn more: [Session Management](./docs/en/concepts/08-session.md) --- ## Project Architecture -The OpenViking project adopts a clear modular architecture design. The main directory structure is as follows: - ``` OpenViking/ -├── openviking/ # Core source code directory -│ ├── core/ # Core modules: client, engine, filesystem, etc. -│ ├── models/ # Model integration: VLM and Embedding model encapsulation -│ ├── parse/ # Resource parsing: file parsing, detection, OVPack handling -│ ├── retrieve/ # Retrieval module: semantic retrieval, directory recursive retrieval -│ ├── storage/ # Storage layer: vector DB, filesystem queue, observers -│ ├── session/ # Session management: history, memory extraction -│ ├── message/ # Message processing: formatting, conversion -│ ├── prompts/ # Prompt templates: templates for various tasks -│ ├── utils/ # Utilities: config, helpers -│ └── bin/ # Command line tools -├── docs/ # Project documentation -│ ├── zh/ # Chinese documentation -│ ├── en/ # English documentation -│ └── images/ # Documentation images -├── examples/ # Usage examples -├── tests/ # Test cases -│ ├── client/ # Client tests -│ ├── engine/ # Engine tests -│ ├── integration/ # Integration tests -│ ├── session/ # Session tests -│ └── vectordb/ # Vector DB tests -├── src/ # C++ extensions (high-performance index and storage) -│ ├── common/ # Common components -│ ├── index/ # Index implementation -│ └── store/ # Storage implementation -├── third_party/ # Third-party dependencies -├── pyproject.toml # Python project configuration -├── setup.py # Setup script -├── LICENSE # Open source license -├── CONTRIBUTING.md # Contributing guide -├── AGENT.md # Agent development guide -└── README.md # Project readme +├── openviking/ # Core source code +│ ├── core/ # Client, engine, filesystem +│ ├── models/ # VLM and Embedding model integration +│ ├── parse/ # Resource parsing +│ ├── retrieve/ # Semantic and directory retrieval +│ ├── storage/ # Vector DB, filesystem queue +│ └── session/ # History, memory extraction +├── docs/ # Documentation +├── examples/ # Usage examples +├── tests/ # Test cases +└── src/ # C++ extensions ``` - ---- - -## Advanced Reading - -For more details, please visit our [Full Documentation](./docs/en/). - --- ## Community & Team ### About Us -OpenViking is an open-source context database initiated and maintained by the **ByteDance Volcengine Viking Team**. - -The Viking team focuses on unstructured information processing and intelligent retrieval, accumulating rich commercial practical experience in context engineering technology: - -- **2019**: VikingDB vector database supported large-scale use across all ByteDance businesses. -- **2023**: VikingDB sold on Volcengine public cloud. -- **2024**: Launched developer product matrix: VikingDB, Viking KnowledgeBase, Viking MemoryBase. -- **2025**: Created upper-layer application products like AI Search and Vaka Knowledge Assistant. -- **Oct 2025**: Open-sourced [MineContext](https://github.com/volcengine/MineContext), exploring proactive AI applications. -- **Jan 2026**: Open-sourced OpenViking, providing underlying context database support for AI Agents. +OpenViking is an open-source context database initiated and maintained by the ByteDance Volcengine Viking Team. -For more details, please see: **[About Us](./docs/en/about/01-about-us.md)** +The Viking team focuses on unstructured information processing and intelligent retrieval: ---- +- **2019**: VikingDB vector database supported large-scale use across all ByteDance businesses +- **2023**: VikingDB sold on Volcengine public cloud +- **2024**: Launched developer product matrix: VikingDB, Viking KnowledgeBase, Viking MemoryBase +- **2025**: Created upper-layer application products like AI Search and Vaka Knowledge Assistant +- **Oct 2025**: Open-sourced [MineContext](https://github.com/volcengine/MineContext) +- **Jan 2026**: Open-sourced OpenViking ### Join the Community -OpenViking is still in its early stages, and there are many areas for improvement and exploration. We sincerely invite every developer passionate about AI Agent technology: - -- Light up a precious **Star** for us to give us the motivation to move forward. -- Visit our [**Website**](https://www.openviking.ai) to understand the philosophy we convey, and use it in your projects via the [**Documentation**](https://www.openviking.ai/docs). Feel the change it brings and give us feedback on your truest experience. -- Join our community to share your insights, help answer others' questions, and jointly create an open and mutually helpful technical atmosphere: - - 📱 **Lark Group**: Scan the QR code to join → [View QR Code](./docs/en/about/01-about-us.md#lark-group) - - 💬 **WeChat Group**: Scan the QR code to add assistant → [View QR Code](./docs/en/about/01-about-us.md#wechat-group) - - 🎮 **Discord**: [Join Discord Server](https://discord.com/invite/eHvx8E9XF3) - - 🐦 **X (Twitter)**:[Follow us](https://x.com/openvikingai) -- Become a **Contributor**, whether submitting a bug fix or contributing a new feature, every line of your code will be an important cornerstone of OpenViking's growth. - -Let's work together to define and build the future of AI Agent context management. The journey has begun, looking forward to your participation! - ---- - -### Star Trend - -[![Star History Chart](https://api.star-history.com/svg?repos=volcengine/OpenViking&type=timeline&legend=top-left)](https://www.star-history.com/#volcengine/OpenViking&type=timeline&legend=top-left) +- 📱 **Lark Group**: [Join here](./docs/en/about/01-about-us.md#lark-group) +- 💬 **WeChat**: [Join here](./docs/en/about/01-about-us.md#wechat-group) +- 🎮 **Discord**: [Join here](https://discord.com/invite/eHvx8E9XF3) +- 🐦 **X (Twitter)**: [Follow us](https://x.com/openvikingai) --- @@ -591,18 +386,16 @@ Let's work together to define and build the future of AI Agent context managemen This project is licensed under the Apache License 2.0 - see the [LICENSE](./LICENSE) file for details. - - - -[release-shield]: https://img.shields.io/github/v/release/volcengine/OpenViking?color=369eff&labelColor=black&logo=github&style=flat-square + +[release-shield]: https://img.shields.io/github/v/release/volcengine/OpenViking?style=flat-square [release-link]: https://github.com/volcengine/OpenViking/releases -[license-shield]: https://img.shields.io/badge/license-apache%202.0-white?labelColor=black&style=flat-square -[license-shield-link]: https://github.com/volcengine/OpenViking/blob/main/LICENSE -[last-commit-shield]: https://img.shields.io/github/last-commit/volcengine/OpenViking?color=c4f042&labelColor=black&style=flat-square -[last-commit-shield-link]: https://github.com/volcengine/OpenViking/commits/main -[github-stars-shield]: https://img.shields.io/github/stars/volcengine/OpenViking?labelColor&style=flat-square&color=ffcb47 -[github-stars-link]: https://github.com/volcengine/OpenViking -[github-issues-shield]: https://img.shields.io/github/issues/volcengine/OpenViking?labelColor=black&style=flat-square&color=ff80eb +[github-stars-shield]: https://img.shields.io/github/stars/volcengine/OpenViking?style=flat-square +[github-stars-link]: https://github.com/volcengine/OpenViking/stargazers +[github-issues-shield]: https://img.shields.io/github/issues/volcengine/OpenViking?style=flat-square [github-issues-shield-link]: https://github.com/volcengine/OpenViking/issues -[github-contributors-shield]: https://img.shields.io/github/contributors/volcengine/OpenViking?color=c4f042&labelColor=black&style=flat-square +[github-contributors-shield]: https://img.shields.io/github/contributors/volcengine/OpenViking?style=flat-square [github-contributors-link]: https://github.com/volcengine/OpenViking/graphs/contributors +[license-shield]: https://img.shields.io/github/license/volcengine/OpenViking?style=flat-square +[license-shield-link]: https://github.com/volcengine/OpenViking/blob/main/LICENSE +[last-commit-shield]: https://img.shields.io/github/last-commit/volcengine/OpenViking?style=flat-square +[last-commit-shield-link]: https://github.com/volcengine/OpenViking/commits/main