From ad510b727b25033b37a8b784836fde7d284c9fce Mon Sep 17 00:00:00 2001 From: Charith Nuwan Bimsara <59943919+nuwangeek@users.noreply.github.com> Date: Wed, 5 Nov 2025 18:52:28 +0530 Subject: [PATCH 1/2] Vault Authentication token handling (#154) * partialy completes prompt refiner * integrate prompt refiner with llm_config_module * fixed ruff lint issues * complete prompt refiner, chunk retriver and reranker * remove unnesessary comments * updated .gitignore * Remove data_sets from tracking * update .gitignore file * complete vault setup and response generator * remove ignore comment * removed old modules * fixed merge conflicts * added initial setup for the vector indexer * initial llm orchestration service update with context generation * added new endpoints * vector indexer with contextual retrieval * fixed requested changes * fixed issue * initial diff identifier setup * uncommment docker compose file * added test endpoint for orchestrate service * fixed ruff linting issue * Rag 103 budget related schema changes (#41) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils --------- Co-authored-by: erangi-ar * Rag 93 update connection status (#47) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Implement LLM connection status update functionality with API integration and UI enhancements --------- Co-authored-by: erangi-ar * Rag 99 production llm connections logic (#46) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Add production connection retrieval and update related components * Implement LLM connection environment update and enhance connection management logic --------- Co-authored-by: erangi-ar * Rag 119 endpoint to update used budget (#42) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add functionality to update used budget for LLM connections with validation and response handling * Implement budget threshold checks and connection deactivation logic in update process * resolve pr comments --------- Co-authored-by: erangi-ar * Rag 113 warning and termination banners (#43) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add budget status check and update BudgetBanner component * rename commonUtils * resove pr comments --------- Co-authored-by: erangi-ar * rag-105-reset-used-budget-cron-job (#44) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add cron job to reset used budget * rename commonUtils * resolve pr comments * Remove trailing slash from vault/agent-out in .gitignore --------- Co-authored-by: erangi-ar * Rag 101 budget check functionality (#45) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * budget check functionality --------- Co-authored-by: erangi-ar * gui running on 3003 issue fixed * gui running on 3003 issue fixed (#50) Co-authored-by: erangi-ar * added get-configuration.sqpl and updated llmconnections.ts * Add SQL query to retrieve configuration values * Hashicorp key saving (#51) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values --------- Co-authored-by: erangi-ar * Remove REACT_APP_NOTIFICATION_NODE_URL variable Removed REACT_APP_NOTIFICATION_NODE_URL environment variable. * added initil diff identifier functionality * test phase1 * Refactor inference and connection handling in YAML and TypeScript files * fixes (#52) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files --------- Co-authored-by: erangi-ar * Add entry point script for Vector Indexer with command line interface * fix (#53) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files * Add entry point script for Vector Indexer with command line interface --------- Co-authored-by: erangi-ar * diff fixes * uncomment llm orchestration service in docker compose file * complete vector indexer * Add YAML configurations and scripts for managing vault secrets * Add vault secret management functions and endpoints for LLM connections * Add Test Production LLM page with messaging functionality and styles * fixed issue * fixed merge conflicts * fixed issue * fixed issue * updated with requested chnages * fixed test ui endpoint request responses schema issue * fixed dvc path issue * added dspy optimization * filters fixed * refactor: restructure llm_connections table for improved configuration and tracking * feat: enhance LLM connection handling with AWS and Azure embedding credentials * fixed issues * refactor: remove redundant Azure and AWS credential assignments in vault secret functions * fixed issue * intial vault setup script * complete vault authentication handling * review requested change fix * fixed issues according to the pr review * fixed issues in docker compose file relevent to pr review --------- Co-authored-by: erangi-ar <111747955+erangi-ar@users.noreply.github.com> Co-authored-by: erangi-ar --- .../script/delete_secrets_from_vault.sh | 183 +++++++++- .../script/store_secrets_in_vault.sh | 314 +++++++++++++++++- .../rag-search/POST/vault/secret/create.yml | 2 +- docker-compose.yml | 55 ++- vault-init.sh | 177 ++++++++++ vault/agents/llm/agent.hcl | 15 +- 6 files changed, 714 insertions(+), 32 deletions(-) create mode 100644 vault-init.sh diff --git a/DSL/CronManager/script/delete_secrets_from_vault.sh b/DSL/CronManager/script/delete_secrets_from_vault.sh index be936dc..86692e3 100644 --- a/DSL/CronManager/script/delete_secrets_from_vault.sh +++ b/DSL/CronManager/script/delete_secrets_from_vault.sh @@ -1,7 +1,182 @@ #!/bin/bash -echo "cookie" -echo $cookie +# Vault Secrets Deletion Script +# This script deletes LLM and embedding credentials from HashiCorp Vault -echo "llmPlatform" -echo $llmPlatform +set -e # Exit on any error + +# Configuration +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +VAULT_TOKEN_FILE="/agent/out/token" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +log "=== Starting Vault Secrets Deletion ===" + +# Debug: Print received parameters +log "Received parameters:" +log " connectionId: $connectionId" +log " llmPlatform: $llmPlatform" +log " llmModel: $llmModel" +log " embeddingModel: $embeddingModel" +log " embeddingPlatform: $embeddingPlatform" +log " deploymentEnvironment: $deploymentEnvironment" + +# Read vault token +if [ ! -f "$VAULT_TOKEN_FILE" ]; then + log "ERROR: Vault token file not found at $VAULT_TOKEN_FILE" + exit 1 +fi + +VAULT_TOKEN=$(cat "$VAULT_TOKEN_FILE") +if [ -z "$VAULT_TOKEN" ]; then + log "ERROR: Vault token is empty" + exit 1 +fi + +log "Vault token loaded successfully" + +# Function to determine platform name +get_platform_name() { + local platform=$1 + case "$platform" in + "aws") echo "aws_bedrock" ;; + "azure") echo "azure_openai" ;; + *) + log "ERROR: Unsupported platform: $platform" + exit 1 + ;; + esac +} + +# Function to get model name (first element from array) +get_model_name() { + local model_array=$1 + # Remove brackets and quotes, get first element + echo "$model_array" | sed 's/\[//g' | sed 's/\]//g' | sed 's/"//g' | cut -d',' -f1 | xargs +} + +# Function to build vault path +build_vault_path() { + local secret_type=$1 # "llm" or "embeddings" + local platform_name=$2 + local model_name=$3 + + if [ "$deploymentEnvironment" = "test" ]; then + echo "secret/$secret_type/connections/$platform_name/$deploymentEnvironment/$connectionId" + else + echo "secret/$secret_type/connections/$platform_name/$deploymentEnvironment/$model_name" + fi +} + +# Function to delete vault secret (both data and metadata) +delete_vault_secret() { + local vault_path=$1 + local secret_description=$2 + + log "Deleting $secret_description at path: $vault_path" + + # Convert path for KV v2 API (secret/path -> secret/data/path and secret/metadata/path) + local data_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + local metadata_path=$(echo "$vault_path" | sed 's|^secret/|secret/metadata/|') + + log "Data API URL: $VAULT_ADDR/v1/$data_path" + log "Metadata API URL: $VAULT_ADDR/v1/$metadata_path" + + local success=true + + # Delete secret data + log "Deleting secret data..." + local data_response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X DELETE \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + "$VAULT_ADDR/v1/$data_path") + + local data_http_code=$(echo "$data_response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local data_body=$(echo "$data_response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$data_http_code" -ge 200 && "$data_http_code" -lt 300 ]] || [[ "$data_http_code" == "404" ]]; then + log "Secret data deleted successfully (HTTP $data_http_code)" + else + log "WARNING: Failed to delete secret data (HTTP $data_http_code)" + log "Data response: $data_body" + success=false + fi + + # Delete secret metadata + log "Deleting secret metadata..." + local metadata_response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X DELETE \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + "$VAULT_ADDR/v1/$metadata_path") + + local metadata_http_code=$(echo "$metadata_response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local metadata_body=$(echo "$metadata_response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$metadata_http_code" -ge 200 && "$metadata_http_code" -lt 300 ]] || [[ "$metadata_http_code" == "404" ]]; then + log "Secret metadata deleted successfully (HTTP $metadata_http_code)" + else + log "WARNING: Failed to delete secret metadata (HTTP $metadata_http_code)" + log "Metadata response: $metadata_body" + success=false + fi + + if [ "$success" = true ]; then + log "$secret_description deletion completed successfully" + else + log "WARNING: $secret_description deletion completed with some failures" + fi + + return 0 # Continue even if some deletions fail +} + +# Function to delete LLM secrets +delete_llm_secrets() { + if [ -z "$llmPlatform" ] || [ -z "$llmModel" ]; then + log "No LLM platform or model specified, skipping LLM secrets deletion" + return 0 + fi + + local platform_name=$(get_platform_name "$llmPlatform") + local model_name=$(get_model_name "$llmModel") + local vault_path=$(build_vault_path "llm" "$platform_name" "$model_name") + + delete_vault_secret "$vault_path" "LLM secrets" +} + +# Function to delete embedding secrets +delete_embedding_secrets() { + if [ -z "$embeddingPlatform" ] || [ -z "$embeddingModel" ]; then + log "No embedding platform or model specified, skipping embedding secrets deletion" + return 0 + fi + + local platform_name=$(get_platform_name "$embeddingPlatform") + local vault_path=$(build_vault_path "embeddings" "$platform_name" "$embeddingModel") + + delete_vault_secret "$vault_path" "Embedding secrets" +} + +# Main execution +if [ -n "$llmPlatform" ]; then + log "LLM Platform: $(get_platform_name "$llmPlatform")" +fi + +if [ -n "$llmModel" ]; then + log "LLM Model: $(get_model_name "$llmModel")" +fi + +if [ -n "$embeddingPlatform" ]; then + log "Embedding Platform: $(get_platform_name "$embeddingPlatform")" +fi + +# Delete LLM secrets +delete_llm_secrets + +# Delete embedding secrets +delete_embedding_secrets + +log "=== Vault secrets deletion completed ===" diff --git a/DSL/CronManager/script/store_secrets_in_vault.sh b/DSL/CronManager/script/store_secrets_in_vault.sh index be936dc..44439b1 100644 --- a/DSL/CronManager/script/store_secrets_in_vault.sh +++ b/DSL/CronManager/script/store_secrets_in_vault.sh @@ -1,7 +1,313 @@ #!/bin/bash -echo "cookie" -echo $cookie +# Vault Secrets Storage Script +# This script stores LLM and embedding credentials in HashiCorp Vault + +set -e # Exit on any error + +# Configuration +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +VAULT_TOKEN_FILE="/agent/out/token" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +log "=== Starting Vault Secrets Storage ===" + +# Debug: Print received parameters +log "Received parameters:" +log " connectionId: $connectionId" +log " llmPlatform: $llmPlatform" +log " llmModel: $llmModel" +log " deploymentEnvironment: $deploymentEnvironment" + +# Read vault token +if [ ! -f "$VAULT_TOKEN_FILE" ]; then + log "ERROR: Vault token file not found at $VAULT_TOKEN_FILE" + exit 1 +fi + +VAULT_TOKEN=$(cat "$VAULT_TOKEN_FILE") +if [ -z "$VAULT_TOKEN" ]; then + log "ERROR: Vault token is empty" + exit 1 +fi + +log "Vault token loaded successfully" + +# Function to determine platform name +get_platform_name() { + case "$llmPlatform" in + "aws") echo "aws_bedrock" ;; + "azure") echo "azure_openai" ;; + *) + log "ERROR: Unsupported platform: $llmPlatform" + exit 1 + ;; + esac +} + +# Function to get model name (first element from array) +get_model_name() { + # Remove brackets and quotes, get first element + echo "$llmModel" | sed 's/\[//g' | sed 's/\]//g' | sed 's/"//g' | cut -d',' -f1 | xargs +} + +# Function to build vault path +build_vault_path() { + local secret_type=$1 # "llm" or "embeddings" + local platform=$(get_platform_name) + local model=$(get_model_name) + + if [ "$deploymentEnvironment" = "test" ]; then + echo "secret/$secret_type/connections/$platform/$deploymentEnvironment/$connectionId" + else + echo "secret/$secret_type/connections/$platform/$deploymentEnvironment/$model" + fi +} + +# Function to store LLM secrets +store_llm_secrets() { + local vault_path=$(build_vault_path "llm") + log "Storing LLM secrets at path: $vault_path" + + case "$llmPlatform" in + "aws") + store_aws_llm_secrets "$vault_path" + ;; + "azure") + store_azure_llm_secrets "$vault_path" + ;; + esac +} + +# Function to store embedding secrets +store_embedding_secrets() { + local vault_path=$(build_vault_path "embeddings") + log "Storing embedding secrets at path: $vault_path" + + case "$embeddingPlatform" in + "aws") + store_aws_embedding_secrets "$vault_path" + ;; + "azure") + store_azure_embedding_secrets "$vault_path" + ;; + *) + log "WARNING: Embedding platform '$embeddingPlatform' not supported, skipping embedding secrets" + ;; + esac +} + +# Function to store AWS LLM secrets +store_aws_llm_secrets() { + local vault_path=$1 + local model=$(get_model_name) + + log "Storing AWS LLM secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "AWS LLM secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store AWS LLM secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store Azure LLM secrets +store_azure_llm_secrets() { + local vault_path=$1 + local model=$(get_model_name) + + log "Storing Azure LLM secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "Azure LLM secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store Azure LLM secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store AWS embedding secrets +store_aws_embedding_secrets() { + local vault_path=$1 + + log "Storing AWS embedding secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "AWS embedding secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store AWS embedding secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store Azure embedding secrets +store_azure_embedding_secrets() { + local vault_path=$1 + + log "Storing Azure embedding secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "Azure embedding secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store Azure embedding secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Main execution +log "Platform: $(get_platform_name)" +log "Model: $(get_model_name)" + +# Store LLM secrets +store_llm_secrets + +# Store embedding secrets if embedding platform is provided +if [ -n "$embeddingPlatform" ]; then + store_embedding_secrets +else + log "No embedding platform specified, skipping embedding secrets" +fi + +log "=== Vault secrets storage completed successfully ===" -echo "llmPlatform" -echo $llmPlatform diff --git a/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml b/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml index 96501b3..3fa2f46 100644 --- a/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml +++ b/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml @@ -12,7 +12,7 @@ declaration: type: string description: "Body field 'connectionId'" - field: llmPlatform - type: number + type: string description: "Body field 'llmPlatform'" - field: llmModel type: array diff --git a/docker-compose.yml b/docker-compose.yml index 5884a96..60cc34c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -184,11 +184,15 @@ services: - ./datasets:/app/datasets # Direct access to datasets folder for diff identifier operations - ./grafana-configs/loki_logger.py:/app/src/vector_indexer/loki_logger.py - ./.env:/app/.env:ro + - vault-agent-token:/agent/out:ro # Mount vault token for accessing vault secrets environment: - server.port=9010 - PYTHONPATH=/app:/app/src/vector_indexer + - VAULT_ADDR=http://vault:8200 ports: - 9010:8080 + depends_on: + - vault-agent-llm networks: - bykstack @@ -439,31 +443,54 @@ services: VAULT_SKIP_VERIFY_CONFIG_PERMISSIONS: "true" volumes: - vault-data:/vault/file - - ./vault/config:/vault/config # contains vault.hcl + - ./vault/config:/vault/config:ro - ./vault/logs:/vault/logs expose: - - "8200" + - "8200" networks: - bykstack restart: unless-stopped healthcheck: - test: ["CMD", "vault", "status", "-format=json"] - interval: 10s - timeout: 5s - retries: 10 + test: ["CMD", "sh", "-c", "wget -q -O- http://127.0.0.1:8200/v1/sys/health || exit 0"] + interval: 5s + timeout: 3s + retries: 20 + start_period: 10s + + vault-init: + image: hashicorp/vault:1.20.3 + container_name: vault-init + user: "0" + depends_on: + vault: + condition: service_healthy + environment: + VAULT_ADDR: http://vault:8200 + volumes: + - vault-data:/vault/data + - vault-agent-creds:/agent/credentials + - vault-agent-token:/agent/out + - ./vault-init.sh:/vault-init.sh:ro + networks: + - bykstack + entrypoint: ["/bin/sh"] + command: ["-c", "apk add --no-cache curl jq && chmod -R 755 /agent/credentials && chmod -R 770 /agent/out && sh /vault-init.sh"] + restart: "no" vault-agent-llm: image: hashicorp/vault:1.20.3 container_name: vault-agent-llm user: "0" - command: ["vault", "agent", "-config=/agent/in/agent.hcl", "-log-level=info"] + command: ["vault", "agent", "-config=/agent/config/agent.hcl", "-log-level=info"] depends_on: - - vault + vault-init: + condition: service_completed_successfully cap_add: - IPC_LOCK volumes: - - ./vault/agents/llm:/agent/in:ro # agent.hcl, role_id, secret_id - - ./vault/agent-out:/agent/out # token output + - ./vault/agents/llm/agent.hcl:/agent/config/agent.hcl:ro + - vault-agent-creds:/agent/credentials:ro + - vault-agent-token:/agent/out networks: - bykstack restart: unless-stopped @@ -488,7 +515,7 @@ services: - ./src/llm_config_module/config:/app/src/llm_config_module/config:ro # Mount logs directory for persistence - llm_orchestration_logs:/app/logs - - ./vault/agent-out:/agent/out:ro + - vault-agent-token:/agent/out:ro networks: - bykstack depends_on: @@ -520,12 +547,14 @@ volumes: name: minio_data vault-data: name: vault-data - vault-agent-out: - name: vault-agent-out shared-volume: name: shared-volume cron_data: name: cron_data + vault-agent-creds: + name: vault-agent-creds + vault-agent-token: + name: vault-agent-token networks: bykstack: diff --git a/vault-init.sh b/vault-init.sh new file mode 100644 index 0000000..cd36e2d --- /dev/null +++ b/vault-init.sh @@ -0,0 +1,177 @@ +#!/bin/sh +set -e + +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +UNSEAL_KEYS_FILE="/vault/data/unseal-keys.json" +INIT_FLAG="/vault/data/.initialized" + +echo "=== Vault Initialization Script ===" + +# Wait for Vault to be ready +echo "Waiting for Vault..." +for i in $(seq 1 30); do + if wget -q -O- "$VAULT_ADDR/v1/sys/health" >/dev/null 2>&1; then + echo "Vault is ready" + break + fi + echo "Waiting... ($i/30)" + sleep 2 +done + +# Check if this is first time +if [ ! -f "$INIT_FLAG" ]; then + echo "=== FIRST TIME DEPLOYMENT ===" + + # Initialize Vault + echo "Initializing Vault..." + wget -q -O- --post-data='{"secret_shares":5,"secret_threshold":3}' \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/init" > "$UNSEAL_KEYS_FILE" + + ROOT_TOKEN=$(grep -o '"root_token":"[^"]*"' "$UNSEAL_KEYS_FILE" | cut -d':' -f2 | tr -d '"') + export VAULT_TOKEN="$ROOT_TOKEN" + + # Extract unseal keys + KEY1=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '2p' | tr -d '"') + KEY2=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '3p' | tr -d '"') + KEY3=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '4p' | tr -d '"') + + # Unseal Vault + echo "Unsealing Vault..." + wget -q -O- --post-data="{\"key\":\"$KEY1\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY2\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY3\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + sleep 2 + echo "Vault unsealed" + + # Enable KV v2 + echo "Enabling KV v2 secrets engine..." + wget -q -O- --post-data='{"type":"kv","options":{"version":"2"}}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/mounts/secret" >/dev/null 2>&1 || echo "KV already enabled" + + # Enable AppRole + echo "Enabling AppRole..." + wget -q -O- --post-data='{"type":"approle"}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/auth/approle" >/dev/null 2>&1 || echo "AppRole already enabled" + + # Create policy + echo "Creating llm-orchestration policy..." + POLICY='path "secret/metadata/llm/*" { capabilities = ["list", "delete"] } +path "secret/data/llm/*" { capabilities = ["create", "read", "update", "delete"] } +path "auth/token/lookup-self" { capabilities = ["read"] } +path "secret/metadata/embeddings/*" { capabilities = ["list", "delete"] } +path "secret/data/embeddings/*" { capabilities = ["create", "read", "update", "delete"] }' + + POLICY_JSON=$(echo "$POLICY" | jq -Rs '{"policy":.}') + wget -q -O- --post-data="$POLICY_JSON" \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/policies/acl/llm-orchestration" >/dev/null + + # Create AppRole + echo "Creating llm-orchestration-service AppRole..." + wget -q -O- --post-data='{"token_policies":["llm-orchestration"],"token_no_default_policy":true,"token_ttl":"1h","token_max_ttl":"24h","secret_id_ttl":"24h","secret_id_num_uses":0,"bind_secret_id":true}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service" >/dev/null + + # Ensure credentials directory exists + mkdir -p /agent/credentials + + # Get role_id + echo "Getting role_id..." + ROLE_ID=$(wget -q -O- \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/role-id" | \ + grep -o '"role_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$ROLE_ID" > /agent/credentials/role_id + + # Generate secret_id + echo "Generating secret_id..." + SECRET_ID=$(wget -q -O- --post-data='' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/secret-id" | \ + grep -o '"secret_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$SECRET_ID" > /agent/credentials/secret_id + + chmod 644 /agent/credentials/role_id /agent/credentials/secret_id + + # Mark as initialized + touch "$INIT_FLAG" + echo "=== First time setup complete ===" + +else + echo "=== SUBSEQUENT DEPLOYMENT ===" + + # Check if Vault is sealed + SEALED=$(wget -q -O- "$VAULT_ADDR/v1/sys/seal-status" | grep -o '"sealed":[^,}]*' | cut -d':' -f2) + + if [ "$SEALED" = "true" ]; then + echo "Vault is sealed. Unsealing..." + + # Load unseal keys + KEY1=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '2p' | tr -d '"') + KEY2=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '3p' | tr -d '"') + KEY3=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '4p' | tr -d '"') + + wget -q -O- --post-data="{\"key\":\"$KEY1\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY2\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY3\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + sleep 2 + echo "Vault unsealed" + + # Get root token + ROOT_TOKEN=$(grep -o '"root_token":"[^"]*"' "$UNSEAL_KEYS_FILE" | cut -d':' -f2 | tr -d '"') + export VAULT_TOKEN="$ROOT_TOKEN" + + # Ensure credentials directory exists + mkdir -p /agent/credentials + + # Regenerate secret_id after unseal + echo "Regenerating secret_id..." + SECRET_ID=$(wget -q -O- --post-data='' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/secret-id" | \ + grep -o '"secret_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$SECRET_ID" > /agent/credentials/secret_id + chmod 644 /agent/credentials/secret_id + + # Ensure role_id exists + if [ ! -f /agent/credentials/role_id ]; then + echo "Copying role_id..." + mkdir -p /agent/credentials + ROLE_ID=$(wget -q -O- \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/role-id" | \ + grep -o '"role_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$ROLE_ID" > /agent/credentials/role_id + chmod 644 /agent/credentials/role_id + fi + else + echo "Vault is unsealed. No action needed." + fi +fi + +echo "=== Vault init complete ===" \ No newline at end of file diff --git a/vault/agents/llm/agent.hcl b/vault/agents/llm/agent.hcl index 7615ce0..4a0b410 100644 --- a/vault/agents/llm/agent.hcl +++ b/vault/agents/llm/agent.hcl @@ -8,8 +8,8 @@ auto_auth { method "approle" { mount_path = "auth/approle" config = { - role_id_file_path = "/agent/in/role_id" - secret_id_file_path = "/agent/in/secret_id" + role_id_file_path = "/agent/credentials/role_id" + secret_id_file_path = "/agent/credentials/secret_id" remove_secret_id_file_after_reading = false } } @@ -30,13 +30,8 @@ listener "tcp" { tls_disable = true } -template { - source = "/dev/null" - destination = "/agent/out/dummy" -} - api_proxy { use_auto_auth_token = true - enforce_consistency = "always" # Strict consistency - when_inconsistent = "forward" # Forward to Vault if inconsistent -} \ No newline at end of file + enforce_consistency = "always" + when_inconsistent = "forward" +} From caf50af197d6a16df09166fc53166138a484bba7 Mon Sep 17 00:00:00 2001 From: erangi-ar <111747955+erangi-ar@users.noreply.github.com> Date: Thu, 6 Nov 2025 16:42:17 +0530 Subject: [PATCH 2/2] Vault Authentication token handling (#154) (#70) * partialy completes prompt refiner * integrate prompt refiner with llm_config_module * fixed ruff lint issues * complete prompt refiner, chunk retriver and reranker * remove unnesessary comments * updated .gitignore * Remove data_sets from tracking * update .gitignore file * complete vault setup and response generator * remove ignore comment * removed old modules * fixed merge conflicts * added initial setup for the vector indexer * initial llm orchestration service update with context generation * added new endpoints * vector indexer with contextual retrieval * fixed requested changes * fixed issue * initial diff identifier setup * uncommment docker compose file * added test endpoint for orchestrate service * fixed ruff linting issue * Rag 103 budget related schema changes (#41) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils --------- * Rag 93 update connection status (#47) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Implement LLM connection status update functionality with API integration and UI enhancements --------- * Rag 99 production llm connections logic (#46) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Add production connection retrieval and update related components * Implement LLM connection environment update and enhance connection management logic --------- * Rag 119 endpoint to update used budget (#42) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add functionality to update used budget for LLM connections with validation and response handling * Implement budget threshold checks and connection deactivation logic in update process * resolve pr comments --------- * Rag 113 warning and termination banners (#43) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add budget status check and update BudgetBanner component * rename commonUtils * resove pr comments --------- * rag-105-reset-used-budget-cron-job (#44) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add cron job to reset used budget * rename commonUtils * resolve pr comments * Remove trailing slash from vault/agent-out in .gitignore --------- * Rag 101 budget check functionality (#45) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * budget check functionality --------- * gui running on 3003 issue fixed * gui running on 3003 issue fixed (#50) * added get-configuration.sqpl and updated llmconnections.ts * Add SQL query to retrieve configuration values * Hashicorp key saving (#51) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values --------- * Remove REACT_APP_NOTIFICATION_NODE_URL variable Removed REACT_APP_NOTIFICATION_NODE_URL environment variable. * added initil diff identifier functionality * test phase1 * Refactor inference and connection handling in YAML and TypeScript files * fixes (#52) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files --------- * Add entry point script for Vector Indexer with command line interface * fix (#53) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files * Add entry point script for Vector Indexer with command line interface --------- * diff fixes * uncomment llm orchestration service in docker compose file * complete vector indexer * Add YAML configurations and scripts for managing vault secrets * Add vault secret management functions and endpoints for LLM connections * Add Test Production LLM page with messaging functionality and styles * fixed issue * fixed merge conflicts * fixed issue * fixed issue * updated with requested chnages * fixed test ui endpoint request responses schema issue * fixed dvc path issue * added dspy optimization * filters fixed * refactor: restructure llm_connections table for improved configuration and tracking * feat: enhance LLM connection handling with AWS and Azure embedding credentials * fixed issues * refactor: remove redundant Azure and AWS credential assignments in vault secret functions * fixed issue * intial vault setup script * complete vault authentication handling * review requested change fix * fixed issues according to the pr review * fixed issues in docker compose file relevent to pr review --------- Co-authored-by: Charith Nuwan Bimsara <59943919+nuwangeek@users.noreply.github.com> Co-authored-by: erangi-ar --- .../script/delete_secrets_from_vault.sh | 183 +++++++++- .../script/store_secrets_in_vault.sh | 314 +++++++++++++++++- .../rag-search/POST/vault/secret/create.yml | 2 +- docker-compose.yml | 55 ++- vault-init.sh | 177 ++++++++++ vault/agents/llm/agent.hcl | 15 +- 6 files changed, 714 insertions(+), 32 deletions(-) create mode 100644 vault-init.sh diff --git a/DSL/CronManager/script/delete_secrets_from_vault.sh b/DSL/CronManager/script/delete_secrets_from_vault.sh index be936dc..86692e3 100644 --- a/DSL/CronManager/script/delete_secrets_from_vault.sh +++ b/DSL/CronManager/script/delete_secrets_from_vault.sh @@ -1,7 +1,182 @@ #!/bin/bash -echo "cookie" -echo $cookie +# Vault Secrets Deletion Script +# This script deletes LLM and embedding credentials from HashiCorp Vault -echo "llmPlatform" -echo $llmPlatform +set -e # Exit on any error + +# Configuration +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +VAULT_TOKEN_FILE="/agent/out/token" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +log "=== Starting Vault Secrets Deletion ===" + +# Debug: Print received parameters +log "Received parameters:" +log " connectionId: $connectionId" +log " llmPlatform: $llmPlatform" +log " llmModel: $llmModel" +log " embeddingModel: $embeddingModel" +log " embeddingPlatform: $embeddingPlatform" +log " deploymentEnvironment: $deploymentEnvironment" + +# Read vault token +if [ ! -f "$VAULT_TOKEN_FILE" ]; then + log "ERROR: Vault token file not found at $VAULT_TOKEN_FILE" + exit 1 +fi + +VAULT_TOKEN=$(cat "$VAULT_TOKEN_FILE") +if [ -z "$VAULT_TOKEN" ]; then + log "ERROR: Vault token is empty" + exit 1 +fi + +log "Vault token loaded successfully" + +# Function to determine platform name +get_platform_name() { + local platform=$1 + case "$platform" in + "aws") echo "aws_bedrock" ;; + "azure") echo "azure_openai" ;; + *) + log "ERROR: Unsupported platform: $platform" + exit 1 + ;; + esac +} + +# Function to get model name (first element from array) +get_model_name() { + local model_array=$1 + # Remove brackets and quotes, get first element + echo "$model_array" | sed 's/\[//g' | sed 's/\]//g' | sed 's/"//g' | cut -d',' -f1 | xargs +} + +# Function to build vault path +build_vault_path() { + local secret_type=$1 # "llm" or "embeddings" + local platform_name=$2 + local model_name=$3 + + if [ "$deploymentEnvironment" = "test" ]; then + echo "secret/$secret_type/connections/$platform_name/$deploymentEnvironment/$connectionId" + else + echo "secret/$secret_type/connections/$platform_name/$deploymentEnvironment/$model_name" + fi +} + +# Function to delete vault secret (both data and metadata) +delete_vault_secret() { + local vault_path=$1 + local secret_description=$2 + + log "Deleting $secret_description at path: $vault_path" + + # Convert path for KV v2 API (secret/path -> secret/data/path and secret/metadata/path) + local data_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + local metadata_path=$(echo "$vault_path" | sed 's|^secret/|secret/metadata/|') + + log "Data API URL: $VAULT_ADDR/v1/$data_path" + log "Metadata API URL: $VAULT_ADDR/v1/$metadata_path" + + local success=true + + # Delete secret data + log "Deleting secret data..." + local data_response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X DELETE \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + "$VAULT_ADDR/v1/$data_path") + + local data_http_code=$(echo "$data_response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local data_body=$(echo "$data_response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$data_http_code" -ge 200 && "$data_http_code" -lt 300 ]] || [[ "$data_http_code" == "404" ]]; then + log "Secret data deleted successfully (HTTP $data_http_code)" + else + log "WARNING: Failed to delete secret data (HTTP $data_http_code)" + log "Data response: $data_body" + success=false + fi + + # Delete secret metadata + log "Deleting secret metadata..." + local metadata_response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X DELETE \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + "$VAULT_ADDR/v1/$metadata_path") + + local metadata_http_code=$(echo "$metadata_response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local metadata_body=$(echo "$metadata_response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$metadata_http_code" -ge 200 && "$metadata_http_code" -lt 300 ]] || [[ "$metadata_http_code" == "404" ]]; then + log "Secret metadata deleted successfully (HTTP $metadata_http_code)" + else + log "WARNING: Failed to delete secret metadata (HTTP $metadata_http_code)" + log "Metadata response: $metadata_body" + success=false + fi + + if [ "$success" = true ]; then + log "$secret_description deletion completed successfully" + else + log "WARNING: $secret_description deletion completed with some failures" + fi + + return 0 # Continue even if some deletions fail +} + +# Function to delete LLM secrets +delete_llm_secrets() { + if [ -z "$llmPlatform" ] || [ -z "$llmModel" ]; then + log "No LLM platform or model specified, skipping LLM secrets deletion" + return 0 + fi + + local platform_name=$(get_platform_name "$llmPlatform") + local model_name=$(get_model_name "$llmModel") + local vault_path=$(build_vault_path "llm" "$platform_name" "$model_name") + + delete_vault_secret "$vault_path" "LLM secrets" +} + +# Function to delete embedding secrets +delete_embedding_secrets() { + if [ -z "$embeddingPlatform" ] || [ -z "$embeddingModel" ]; then + log "No embedding platform or model specified, skipping embedding secrets deletion" + return 0 + fi + + local platform_name=$(get_platform_name "$embeddingPlatform") + local vault_path=$(build_vault_path "embeddings" "$platform_name" "$embeddingModel") + + delete_vault_secret "$vault_path" "Embedding secrets" +} + +# Main execution +if [ -n "$llmPlatform" ]; then + log "LLM Platform: $(get_platform_name "$llmPlatform")" +fi + +if [ -n "$llmModel" ]; then + log "LLM Model: $(get_model_name "$llmModel")" +fi + +if [ -n "$embeddingPlatform" ]; then + log "Embedding Platform: $(get_platform_name "$embeddingPlatform")" +fi + +# Delete LLM secrets +delete_llm_secrets + +# Delete embedding secrets +delete_embedding_secrets + +log "=== Vault secrets deletion completed ===" diff --git a/DSL/CronManager/script/store_secrets_in_vault.sh b/DSL/CronManager/script/store_secrets_in_vault.sh index be936dc..44439b1 100644 --- a/DSL/CronManager/script/store_secrets_in_vault.sh +++ b/DSL/CronManager/script/store_secrets_in_vault.sh @@ -1,7 +1,313 @@ #!/bin/bash -echo "cookie" -echo $cookie +# Vault Secrets Storage Script +# This script stores LLM and embedding credentials in HashiCorp Vault + +set -e # Exit on any error + +# Configuration +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +VAULT_TOKEN_FILE="/agent/out/token" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" +} + +log "=== Starting Vault Secrets Storage ===" + +# Debug: Print received parameters +log "Received parameters:" +log " connectionId: $connectionId" +log " llmPlatform: $llmPlatform" +log " llmModel: $llmModel" +log " deploymentEnvironment: $deploymentEnvironment" + +# Read vault token +if [ ! -f "$VAULT_TOKEN_FILE" ]; then + log "ERROR: Vault token file not found at $VAULT_TOKEN_FILE" + exit 1 +fi + +VAULT_TOKEN=$(cat "$VAULT_TOKEN_FILE") +if [ -z "$VAULT_TOKEN" ]; then + log "ERROR: Vault token is empty" + exit 1 +fi + +log "Vault token loaded successfully" + +# Function to determine platform name +get_platform_name() { + case "$llmPlatform" in + "aws") echo "aws_bedrock" ;; + "azure") echo "azure_openai" ;; + *) + log "ERROR: Unsupported platform: $llmPlatform" + exit 1 + ;; + esac +} + +# Function to get model name (first element from array) +get_model_name() { + # Remove brackets and quotes, get first element + echo "$llmModel" | sed 's/\[//g' | sed 's/\]//g' | sed 's/"//g' | cut -d',' -f1 | xargs +} + +# Function to build vault path +build_vault_path() { + local secret_type=$1 # "llm" or "embeddings" + local platform=$(get_platform_name) + local model=$(get_model_name) + + if [ "$deploymentEnvironment" = "test" ]; then + echo "secret/$secret_type/connections/$platform/$deploymentEnvironment/$connectionId" + else + echo "secret/$secret_type/connections/$platform/$deploymentEnvironment/$model" + fi +} + +# Function to store LLM secrets +store_llm_secrets() { + local vault_path=$(build_vault_path "llm") + log "Storing LLM secrets at path: $vault_path" + + case "$llmPlatform" in + "aws") + store_aws_llm_secrets "$vault_path" + ;; + "azure") + store_azure_llm_secrets "$vault_path" + ;; + esac +} + +# Function to store embedding secrets +store_embedding_secrets() { + local vault_path=$(build_vault_path "embeddings") + log "Storing embedding secrets at path: $vault_path" + + case "$embeddingPlatform" in + "aws") + store_aws_embedding_secrets "$vault_path" + ;; + "azure") + store_azure_embedding_secrets "$vault_path" + ;; + *) + log "WARNING: Embedding platform '$embeddingPlatform' not supported, skipping embedding secrets" + ;; + esac +} + +# Function to store AWS LLM secrets +store_aws_llm_secrets() { + local vault_path=$1 + local model=$(get_model_name) + + log "Storing AWS LLM secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "AWS LLM secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store AWS LLM secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store Azure LLM secrets +store_azure_llm_secrets() { + local vault_path=$1 + local model=$(get_model_name) + + log "Storing Azure LLM secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "Azure LLM secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store Azure LLM secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store AWS embedding secrets +store_aws_embedding_secrets() { + local vault_path=$1 + + log "Storing AWS embedding secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "AWS embedding secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store AWS embedding secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Function to store Azure embedding secrets +store_azure_embedding_secrets() { + local vault_path=$1 + + log "Storing Azure embedding secrets..." + + # Build JSON payload + local json_payload=$(cat < secret/data/path) + local api_path=$(echo "$vault_path" | sed 's|^secret/|secret/data/|') + log "API URL: $VAULT_ADDR/v1/$api_path" + + # Execute HTTP API call + local response=$(curl -s -w "HTTPSTATUS:%{http_code}" \ + -X POST \ + -H "X-Vault-Token: $VAULT_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$json_payload" \ + "$VAULT_ADDR/v1/$api_path") + + local http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2) + local body=$(echo "$response" | sed -E 's/HTTPSTATUS:[0-9]*$//') + + if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then + log "Azure embedding secrets stored successfully (HTTP $http_code)" + else + log "ERROR: Failed to store Azure embedding secrets (HTTP $http_code)" + log "Response: $body" + exit 1 + fi +} + +# Main execution +log "Platform: $(get_platform_name)" +log "Model: $(get_model_name)" + +# Store LLM secrets +store_llm_secrets + +# Store embedding secrets if embedding platform is provided +if [ -n "$embeddingPlatform" ]; then + store_embedding_secrets +else + log "No embedding platform specified, skipping embedding secrets" +fi + +log "=== Vault secrets storage completed successfully ===" -echo "llmPlatform" -echo $llmPlatform diff --git a/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml b/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml index 96501b3..3fa2f46 100644 --- a/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml +++ b/DSL/Ruuter.private/rag-search/POST/vault/secret/create.yml @@ -12,7 +12,7 @@ declaration: type: string description: "Body field 'connectionId'" - field: llmPlatform - type: number + type: string description: "Body field 'llmPlatform'" - field: llmModel type: array diff --git a/docker-compose.yml b/docker-compose.yml index 5884a96..60cc34c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -184,11 +184,15 @@ services: - ./datasets:/app/datasets # Direct access to datasets folder for diff identifier operations - ./grafana-configs/loki_logger.py:/app/src/vector_indexer/loki_logger.py - ./.env:/app/.env:ro + - vault-agent-token:/agent/out:ro # Mount vault token for accessing vault secrets environment: - server.port=9010 - PYTHONPATH=/app:/app/src/vector_indexer + - VAULT_ADDR=http://vault:8200 ports: - 9010:8080 + depends_on: + - vault-agent-llm networks: - bykstack @@ -439,31 +443,54 @@ services: VAULT_SKIP_VERIFY_CONFIG_PERMISSIONS: "true" volumes: - vault-data:/vault/file - - ./vault/config:/vault/config # contains vault.hcl + - ./vault/config:/vault/config:ro - ./vault/logs:/vault/logs expose: - - "8200" + - "8200" networks: - bykstack restart: unless-stopped healthcheck: - test: ["CMD", "vault", "status", "-format=json"] - interval: 10s - timeout: 5s - retries: 10 + test: ["CMD", "sh", "-c", "wget -q -O- http://127.0.0.1:8200/v1/sys/health || exit 0"] + interval: 5s + timeout: 3s + retries: 20 + start_period: 10s + + vault-init: + image: hashicorp/vault:1.20.3 + container_name: vault-init + user: "0" + depends_on: + vault: + condition: service_healthy + environment: + VAULT_ADDR: http://vault:8200 + volumes: + - vault-data:/vault/data + - vault-agent-creds:/agent/credentials + - vault-agent-token:/agent/out + - ./vault-init.sh:/vault-init.sh:ro + networks: + - bykstack + entrypoint: ["/bin/sh"] + command: ["-c", "apk add --no-cache curl jq && chmod -R 755 /agent/credentials && chmod -R 770 /agent/out && sh /vault-init.sh"] + restart: "no" vault-agent-llm: image: hashicorp/vault:1.20.3 container_name: vault-agent-llm user: "0" - command: ["vault", "agent", "-config=/agent/in/agent.hcl", "-log-level=info"] + command: ["vault", "agent", "-config=/agent/config/agent.hcl", "-log-level=info"] depends_on: - - vault + vault-init: + condition: service_completed_successfully cap_add: - IPC_LOCK volumes: - - ./vault/agents/llm:/agent/in:ro # agent.hcl, role_id, secret_id - - ./vault/agent-out:/agent/out # token output + - ./vault/agents/llm/agent.hcl:/agent/config/agent.hcl:ro + - vault-agent-creds:/agent/credentials:ro + - vault-agent-token:/agent/out networks: - bykstack restart: unless-stopped @@ -488,7 +515,7 @@ services: - ./src/llm_config_module/config:/app/src/llm_config_module/config:ro # Mount logs directory for persistence - llm_orchestration_logs:/app/logs - - ./vault/agent-out:/agent/out:ro + - vault-agent-token:/agent/out:ro networks: - bykstack depends_on: @@ -520,12 +547,14 @@ volumes: name: minio_data vault-data: name: vault-data - vault-agent-out: - name: vault-agent-out shared-volume: name: shared-volume cron_data: name: cron_data + vault-agent-creds: + name: vault-agent-creds + vault-agent-token: + name: vault-agent-token networks: bykstack: diff --git a/vault-init.sh b/vault-init.sh new file mode 100644 index 0000000..cd36e2d --- /dev/null +++ b/vault-init.sh @@ -0,0 +1,177 @@ +#!/bin/sh +set -e + +VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}" +UNSEAL_KEYS_FILE="/vault/data/unseal-keys.json" +INIT_FLAG="/vault/data/.initialized" + +echo "=== Vault Initialization Script ===" + +# Wait for Vault to be ready +echo "Waiting for Vault..." +for i in $(seq 1 30); do + if wget -q -O- "$VAULT_ADDR/v1/sys/health" >/dev/null 2>&1; then + echo "Vault is ready" + break + fi + echo "Waiting... ($i/30)" + sleep 2 +done + +# Check if this is first time +if [ ! -f "$INIT_FLAG" ]; then + echo "=== FIRST TIME DEPLOYMENT ===" + + # Initialize Vault + echo "Initializing Vault..." + wget -q -O- --post-data='{"secret_shares":5,"secret_threshold":3}' \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/init" > "$UNSEAL_KEYS_FILE" + + ROOT_TOKEN=$(grep -o '"root_token":"[^"]*"' "$UNSEAL_KEYS_FILE" | cut -d':' -f2 | tr -d '"') + export VAULT_TOKEN="$ROOT_TOKEN" + + # Extract unseal keys + KEY1=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '2p' | tr -d '"') + KEY2=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '3p' | tr -d '"') + KEY3=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '4p' | tr -d '"') + + # Unseal Vault + echo "Unsealing Vault..." + wget -q -O- --post-data="{\"key\":\"$KEY1\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY2\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY3\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + sleep 2 + echo "Vault unsealed" + + # Enable KV v2 + echo "Enabling KV v2 secrets engine..." + wget -q -O- --post-data='{"type":"kv","options":{"version":"2"}}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/mounts/secret" >/dev/null 2>&1 || echo "KV already enabled" + + # Enable AppRole + echo "Enabling AppRole..." + wget -q -O- --post-data='{"type":"approle"}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/auth/approle" >/dev/null 2>&1 || echo "AppRole already enabled" + + # Create policy + echo "Creating llm-orchestration policy..." + POLICY='path "secret/metadata/llm/*" { capabilities = ["list", "delete"] } +path "secret/data/llm/*" { capabilities = ["create", "read", "update", "delete"] } +path "auth/token/lookup-self" { capabilities = ["read"] } +path "secret/metadata/embeddings/*" { capabilities = ["list", "delete"] } +path "secret/data/embeddings/*" { capabilities = ["create", "read", "update", "delete"] }' + + POLICY_JSON=$(echo "$POLICY" | jq -Rs '{"policy":.}') + wget -q -O- --post-data="$POLICY_JSON" \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/policies/acl/llm-orchestration" >/dev/null + + # Create AppRole + echo "Creating llm-orchestration-service AppRole..." + wget -q -O- --post-data='{"token_policies":["llm-orchestration"],"token_no_default_policy":true,"token_ttl":"1h","token_max_ttl":"24h","secret_id_ttl":"24h","secret_id_num_uses":0,"bind_secret_id":true}' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service" >/dev/null + + # Ensure credentials directory exists + mkdir -p /agent/credentials + + # Get role_id + echo "Getting role_id..." + ROLE_ID=$(wget -q -O- \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/role-id" | \ + grep -o '"role_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$ROLE_ID" > /agent/credentials/role_id + + # Generate secret_id + echo "Generating secret_id..." + SECRET_ID=$(wget -q -O- --post-data='' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/secret-id" | \ + grep -o '"secret_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$SECRET_ID" > /agent/credentials/secret_id + + chmod 644 /agent/credentials/role_id /agent/credentials/secret_id + + # Mark as initialized + touch "$INIT_FLAG" + echo "=== First time setup complete ===" + +else + echo "=== SUBSEQUENT DEPLOYMENT ===" + + # Check if Vault is sealed + SEALED=$(wget -q -O- "$VAULT_ADDR/v1/sys/seal-status" | grep -o '"sealed":[^,}]*' | cut -d':' -f2) + + if [ "$SEALED" = "true" ]; then + echo "Vault is sealed. Unsealing..." + + # Load unseal keys + KEY1=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '2p' | tr -d '"') + KEY2=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '3p' | tr -d '"') + KEY3=$(grep -o '"keys":\[[^]]*\]' "$UNSEAL_KEYS_FILE" | grep -o '"[^"]*"' | sed -n '4p' | tr -d '"') + + wget -q -O- --post-data="{\"key\":\"$KEY1\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY2\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + wget -q -O- --post-data="{\"key\":\"$KEY3\"}" \ + --header='Content-Type: application/json' \ + "$VAULT_ADDR/v1/sys/unseal" >/dev/null + + sleep 2 + echo "Vault unsealed" + + # Get root token + ROOT_TOKEN=$(grep -o '"root_token":"[^"]*"' "$UNSEAL_KEYS_FILE" | cut -d':' -f2 | tr -d '"') + export VAULT_TOKEN="$ROOT_TOKEN" + + # Ensure credentials directory exists + mkdir -p /agent/credentials + + # Regenerate secret_id after unseal + echo "Regenerating secret_id..." + SECRET_ID=$(wget -q -O- --post-data='' \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/secret-id" | \ + grep -o '"secret_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$SECRET_ID" > /agent/credentials/secret_id + chmod 644 /agent/credentials/secret_id + + # Ensure role_id exists + if [ ! -f /agent/credentials/role_id ]; then + echo "Copying role_id..." + mkdir -p /agent/credentials + ROLE_ID=$(wget -q -O- \ + --header="X-Vault-Token: $ROOT_TOKEN" \ + "$VAULT_ADDR/v1/auth/approle/role/llm-orchestration-service/role-id" | \ + grep -o '"role_id":"[^"]*"' | cut -d':' -f2 | tr -d '"') + echo "$ROLE_ID" > /agent/credentials/role_id + chmod 644 /agent/credentials/role_id + fi + else + echo "Vault is unsealed. No action needed." + fi +fi + +echo "=== Vault init complete ===" \ No newline at end of file diff --git a/vault/agents/llm/agent.hcl b/vault/agents/llm/agent.hcl index 7615ce0..4a0b410 100644 --- a/vault/agents/llm/agent.hcl +++ b/vault/agents/llm/agent.hcl @@ -8,8 +8,8 @@ auto_auth { method "approle" { mount_path = "auth/approle" config = { - role_id_file_path = "/agent/in/role_id" - secret_id_file_path = "/agent/in/secret_id" + role_id_file_path = "/agent/credentials/role_id" + secret_id_file_path = "/agent/credentials/secret_id" remove_secret_id_file_after_reading = false } } @@ -30,13 +30,8 @@ listener "tcp" { tls_disable = true } -template { - source = "/dev/null" - destination = "/agent/out/dummy" -} - api_proxy { use_auto_auth_token = true - enforce_consistency = "always" # Strict consistency - when_inconsistent = "forward" # Forward to Vault if inconsistent -} \ No newline at end of file + enforce_consistency = "always" + when_inconsistent = "forward" +}