Complete langfuse setup (buerokratt#182)

nuwangeek · erangi-ar · erangi-ar · web-flow · commit 97f0583c662e · 2025-12-02T12:59:42.000+05:30
* partialy completes prompt refiner * integrate prompt refiner with llm_config_module * fixed ruff lint issues * complete prompt refiner, chunk retriver and reranker * remove unnesessary comments * updated .gitignore * Remove data_sets from tracking * update .gitignore file * complete vault setup and response generator * remove ignore comment * removed old modules * fixed merge conflicts * Vault Authentication token handling (buerokratt#154) (#70) * partialy completes prompt refiner * integrate prompt refiner with llm_config_module * fixed ruff lint issues * complete prompt refiner, chunk retriver and reranker * remove unnesessary comments * updated .gitignore * Remove data_sets from tracking * update .gitignore file * complete vault setup and response generator * remove ignore comment * removed old modules * fixed merge conflicts * added initial setup for the vector indexer * initial llm orchestration service update with context generation * added new endpoints * vector indexer with contextual retrieval * fixed requested changes * fixed issue * initial diff identifier setup * uncommment docker compose file * added test endpoint for orchestrate service * fixed ruff linting issue * Rag 103 budget related schema changes (#41) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils --------- * Rag 93 update connection status (#47) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Implement LLM connection status update functionality with API integration and UI enhancements --------- * Rag 99 production llm connections logic (#46) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * Add production connection retrieval and update related components * Implement LLM connection environment update and enhance connection management logic --------- * Rag 119 endpoint to update used budget (#42) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add functionality to update used budget for LLM connections with validation and response handling * Implement budget threshold checks and connection deactivation logic in update process * resolve pr comments --------- * Rag 113 warning and termination banners (#43) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add budget status check and update BudgetBanner component * rename commonUtils * resove pr comments --------- * rag-105-reset-used-budget-cron-job (#44) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * Add cron job to reset used budget * rename commonUtils * resolve pr comments * Remove trailing slash from vault/agent-out in .gitignore --------- * Rag 101 budget check functionality (#45) * Refactor llm_connections table: update budget tracking fields and reorder columns * Add budget threshold fields and logic to LLM connection management * Enhance budget management: update budget status logic, adjust thresholds, and improve form handling for LLM connections * resolve pr comments & refactoring * rename commonUtils * budget check functionality --------- * gui running on 3003 issue fixed * gui running on 3003 issue fixed (#50) * added get-configuration.sqpl and updated llmconnections.ts * Add SQL query to retrieve configuration values * Hashicorp key saving (#51) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values --------- * Remove REACT_APP_NOTIFICATION_NODE_URL variable Removed REACT_APP_NOTIFICATION_NODE_URL environment variable. * added initil diff identifier functionality * test phase1 * Refactor inference and connection handling in YAML and TypeScript files * fixes (#52) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files --------- * Add entry point script for Vector Indexer with command line interface * fix (#53) * gui running on 3003 issue fixed * Add SQL query to retrieve configuration values * Refactor inference and connection handling in YAML and TypeScript files * Add entry point script for Vector Indexer with command line interface --------- * diff fixes * uncomment llm orchestration service in docker compose file * complete vector indexer * Add YAML configurations and scripts for managing vault secrets * Add vault secret management functions and endpoints for LLM connections * Add Test Production LLM page with messaging functionality and styles * fixed issue * fixed merge conflicts * fixed issue * fixed issue * updated with requested chnages * fixed test ui endpoint request responses schema issue * fixed dvc path issue * added dspy optimization * filters fixed * refactor: restructure llm_connections table for improved configuration and tracking * feat: enhance LLM connection handling with AWS and Azure embedding credentials * fixed issues * refactor: remove redundant Azure and AWS credential assignments in vault secret functions * fixed issue * intial vault setup script * complete vault authentication handling * review requested change fix * fixed issues according to the pr review * fixed issues in docker compose file relevent to pr review --------- Co-authored-by: Charith Nuwan Bimsara <59943919+nuwangeek@users.noreply.github.com> Co-authored-by: erangi-ar <erangika.ariyasena@rootcode.io> * fixed number chunk issue * fixed ruff format issue * complete inference result update and budget updating * fixed issues * fixed ruff format issue * fixed endpoint issue * fixed format issues * fixed issue * fixed issue * complete langfuse setup * fixed review comments --------- Co-authored-by: erangi-ar <111747955+erangi-ar@users.noreply.github.com> Co-authored-by: erangi-ar <erangika.ariyasena@rootcode.io>
diff --git a/README.md b/README.md
@@ -24,3 +24,19 @@ The **BYK-RAG Module** is part of the Burokratt ecosystem, designed to provide *
   - External **Langfuse dashboard** for API usage, inference trends, cost analysis, and performance logs.  
   - Agencies can configure cost alerts and view alerts via LLM Alerts UI.  
   - Logs integrated with **Grafana Loki**.  
+
+### Storing Langfuse Secrets
+
+1. **Generate API keys from Langfuse UI** (Settings → Project → API Keys)
+
+2. **Copy the script to vault container:**
+```bash
+docker cp store-langfuse-secrets.sh vault:/tmp/store-langfuse-secrets.sh
+```
+
+3. **Execute the script with your API keys:**
+```bash
+docker exec -e LANGFUSE_INIT_PROJECT_PUBLIC_KEY=<your public key> \
+            -e LANGFUSE_INIT_PROJECT_SECRET_KEY=<your secret key> \
+            vault sh -c "chmod +x /tmp/store-langfuse-secrets.sh && /tmp/store-langfuse-secrets.sh"
+```
diff --git a/env.example b/env.example
@@ -6,8 +6,8 @@ S3_ENDPOINT_NAME=minio:9000
 S3_DATA_BUCKET_PATH=resources
 S3_DATA_BUCKET_NAME=rag-search
 FS_DATA_DIRECTORY_PATH=/app
-S3_SECRET_ACCESS_KEY=changeme
-S3_ACCESS_KEY_ID=changeme
+S3_SECRET_ACCESS_KEY=minioadmin
+S3_ACCESS_KEY_ID=minioadmin
 S3_HEALTH_ENDPOINT=http://minio:9000/minio/health/live
 MINIO_BROWSER_REDIRECT_URL=http://localhost:9091
 GF_SECURITY_ADMIN_USER=admin
@@ -16,8 +16,8 @@ GF_USERS_ALLOW_SIGN_UP=false
 PORT=3000
 POSTGRES_USER=postgres
 POSTGRES_PASSWORD=dbadmin
-POSTGRES_DB=rag-search
-NEXTAUTH_URL=http://localhost:3000
+POSTGRES_DB=rag-search-langfuse
+NEXTAUTH_URL=http://localhost:3005
 DATABASE_URL=postgresql://postgres:dbadmin@rag_search_db:5432/rag-search
 SALT=changeme
 ENCRYPTION_KEY=changeme
@@ -27,7 +27,7 @@ LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=true
 CLICKHOUSE_MIGRATION_URL=clickhouse://clickhouse:9000
 CLICKHOUSE_URL=http://clickhouse:8123
 CLICKHOUSE_USER=clickhouse
-CLICKHOUSE_PASSWORD=changeme
+CLICKHOUSE_PASSWORD=clickhouse
 CLICKHOUSE_CLUSTER_ENABLED=false
 LANGFUSE_USE_AZURE_BLOB=false
 LANGFUSE_S3_EVENT_UPLOAD_BUCKET=rag-search
@@ -41,15 +41,15 @@ LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=rag-search
 LANGFUSE_S3_MEDIA_UPLOAD_REGION=auto
 LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=changeme
 LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=changeme
-LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://localhost:9090
+LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://minio:9000
 LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
 LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=langfuse/media/
 LANGFUSE_S3_BATCH_EXPORT_ENABLED=false
 LANGFUSE_S3_BATCH_EXPORT_BUCKET=rag-search
 LANGFUSE_S3_BATCH_EXPORT_PREFIX=langfuse/exports/
 LANGFUSE_S3_BATCH_EXPORT_REGION=auto
 LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=http://minio:9000
-LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://localhost:9090
+LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://minio:9000
 LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=changeme
 LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=changeme
 LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
@@ -64,9 +64,7 @@ REDIS_TLS_CERT=/certs/redis.crt
 REDIS_TLS_KEY=/certs/redis.key
 EMAIL_FROM_ADDRESS=
 SMTP_CONNECTION_URL=
-AZURE_OPENAI_ENDPOINT=your_azure_openai_endpoint_here
-AZURE_OPENAI_API_KEY=your_azure_openai_api_key_here
-AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o-mini
-AWS_REGION=us-east-1
-AWS_ACCESS_KEY_ID=your_aws_access_key_here
-AWS_SECRET_ACCESS_KEY=your_aws_secret_key_here
+VAULT_ADDR=http://localhost:8200
+S3_FERRY_URL=http://rag-s3-ferry:3000/v1/files/copy
+DATASETS_PATH=/app/datasets
+METADATA_FILENAME=processed-metadata.json
diff --git a/src/llm_orchestration_service.py b/src/llm_orchestration_service.py
@@ -1884,7 +1884,7 @@ def _refine_user_prompt(
                 )
             output_json = validated_output.model_dump()
             logger.info(
-                f"Prompt refinement output: {json.dumps(output_json, indent=2)}"
+                f"Prompt refinement output: {json_module.dumps(output_json, indent=2)}"
             )
 
             logger.info("Prompt refinement completed successfully")
diff --git a/src/utils/production_store.py b/src/utils/production_store.py
@@ -12,7 +12,9 @@
 import requests
 import aiohttp
 from src.utils.connection_id_fetcher import get_connection_id_fetcher
-from ..llm_orchestrator_config.llm_ochestrator_constants import RAG_SEARCH_RUUTER_PUBLIC
+from src.llm_orchestrator_config.llm_ochestrator_constants import (
+    RAG_SEARCH_RUUTER_PUBLIC,
+)
 
 
 class ProductionInferenceStore:
diff --git a/store-langfuse-secrets.sh b/store-langfuse-secrets.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+set -e
+
+# ============================================================================
+# Langfuse Secrets Storage Script for Vault
+# ============================================================================
+# This script stores Langfuse configuration secrets in HashiCorp Vault.
+# Run this script AFTER vault-init.sh has completed successfully.
+#
+# Prerequisites:
+# 1. Vault must be initialized and unsealed
+# 2. Environment variables must be set (LANGFUSE_INIT_PROJECT_PUBLIC_KEY, etc.)
+# 3. Root token must be available in /vault/file/unseal-keys.json
+#
+# Usage:
+#   ./store-langfuse-secrets.sh
+#
+# Or with custom values:
+#   LANGFUSE_INIT_PROJECT_PUBLIC_KEY=pk-xxx \
+#   LANGFUSE_INIT_PROJECT_SECRET_KEY=sk-xxx \
+#   LANGFUSE_HOST=http://langfuse-web:3000 \
+#   ./store-langfuse-secrets.sh
+# ============================================================================
+
+VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}"
+UNSEAL_KEYS_FILE="/vault/file/unseal-keys.json"
+
+echo "========================================"
+echo "Langfuse Secrets Storage Script"
+echo "========================================"
+
+# Check if Vault is available
+echo "Checking Vault availability..."
+if ! wget -q -O- "$VAULT_ADDR/v1/sys/health" >/dev/null 2>&1; then
+    echo "Error: Vault is not available at $VAULT_ADDR"
+    echo "  Please ensure Vault is running and accessible."
+    exit 1
+fi
+echo "Vault is available"
+
+# Check if Vault is sealed
+SEALED=$(wget -q -O- "$VAULT_ADDR/v1/sys/seal-status" | grep -o '"sealed":[^,}]*' | cut -d':' -f2)
+if [ "$SEALED" = "true" ]; then
+    echo "Error: Vault is sealed"
+    echo "  Please unseal Vault first using vault-init.sh or manual unseal process."
+    exit 1
+fi
+echo "Vault is unsealed"
+
+# Get root token
+echo "Loading Vault root token..."
+if [ ! -f "$UNSEAL_KEYS_FILE" ]; then
+    echo "Error: Unseal keys file not found at $UNSEAL_KEYS_FILE"
+    echo "  Please run vault-init.sh first to initialize Vault."
+    exit 1
+fi
+
+ROOT_TOKEN=$(grep -o '"root_token":"[^"]*"' "$UNSEAL_KEYS_FILE" | cut -d':' -f2 | tr -d '"')
+if [ -z "$ROOT_TOKEN" ]; then
+    echo "Error: Could not extract root token from $UNSEAL_KEYS_FILE"
+    exit 1
+fi
+echo "Root token loaded"
+
+# Check required environment variables
+echo "Checking Langfuse environment variables..."
+if [ -z "$LANGFUSE_INIT_PROJECT_PUBLIC_KEY" ]; then
+    echo "Error: LANGFUSE_INIT_PROJECT_PUBLIC_KEY is not set"
+    echo "  Please set this environment variable before running the script."
+    echo ""
+    echo "  Example:"
+    echo "    export LANGFUSE_INIT_PROJECT_PUBLIC_KEY='pk-lf-...'"
+    exit 1
+fi
+
+if [ -z "$LANGFUSE_INIT_PROJECT_SECRET_KEY" ]; then
+    echo "Error: LANGFUSE_INIT_PROJECT_SECRET_KEY is not set"
+    echo "  Please set this environment variable before running the script."
+    echo ""
+    echo "  Example:"
+    echo "    export LANGFUSE_INIT_PROJECT_SECRET_KEY='sk-lf-...'"
+    exit 1
+fi
+
+# Use default host if not specified
+LANGFUSE_HOST="${LANGFUSE_HOST:-http://langfuse-web:3000}"
+
+echo "Langfuse environment variables found"
+echo "  Public Key: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:0:10}..."
+echo "  Secret Key: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:0:10}..."
+echo "  Host: $LANGFUSE_HOST"
+
+# Update Vault policy to include Langfuse secrets access
+echo ""
+echo "Updating llm-orchestration policy to include Langfuse secrets..."
+POLICY='path "secret/metadata/llm/*" { capabilities = ["list", "delete"] }
+path "secret/data/llm/*" { capabilities = ["create", "read", "update", "delete"] }
+path "secret/metadata/embeddings/*" { capabilities = ["list", "delete"] }
+path "secret/data/embeddings/*" { capabilities = ["create", "read", "update", "delete"] }
+path "secret/metadata/langfuse/*" { capabilities = ["list", "delete"] }
+path "secret/data/langfuse/*" { capabilities = ["create", "read", "update", "delete"] }
+path "auth/token/lookup-self" { capabilities = ["read"] }'
+
+# Create JSON without jq (using printf for proper escaping)
+POLICY_ESCAPED=$(printf '%s' "$POLICY" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')
+POLICY_JSON='{"policy":"'"$POLICY_ESCAPED"'"}'
+
+if wget -q -O- --post-data="$POLICY_JSON" \
+    --header="X-Vault-Token: $ROOT_TOKEN" \
+    --header='Content-Type: application/json' \
+    "$VAULT_ADDR/v1/sys/policies/acl/llm-orchestration" >/dev/null 2>&1; then
+    echo "Policy updated successfully"
+else
+    echo "Warning: Policy update failed (may already be updated)"
+fi
+
+# Store Langfuse secrets in Vault
+echo ""
+echo "Storing Langfuse secrets in Vault..."
+
+# Create JSON payload
+LANGFUSE_SECRET='{"data":{"public_key":"'"$LANGFUSE_INIT_PROJECT_PUBLIC_KEY"'","secret_key":"'"$LANGFUSE_INIT_PROJECT_SECRET_KEY"'","host":"'"$LANGFUSE_HOST"'"}}'
+
+# Store in Vault
+if wget -q -O- --post-data="$LANGFUSE_SECRET" \
+    --header="X-Vault-Token: $ROOT_TOKEN" \
+    --header='Content-Type: application/json' \
+    "$VAULT_ADDR/v1/secret/data/langfuse/config" >/dev/null 2>&1; then
+    echo "Langfuse secrets stored successfully"
+else
+    echo "Error: Failed to store Langfuse secrets"
+    exit 1
+fi
+
+# Verify secrets were stored
+echo ""
+echo "Verifying stored secrets..."
+VERIFICATION=$(wget -q -O- \
+    --header="X-Vault-Token: $ROOT_TOKEN" \
+    "$VAULT_ADDR/v1/secret/data/langfuse/config" 2>/dev/null)
+
+if echo "$VERIFICATION" | grep -q '"public_key"'; then
+    echo "Secrets verified successfully"
+    echo ""
+    echo "========================================"
+    echo "SUCCESS"
+    echo "========================================"
+    echo "Langfuse secrets have been stored in Vault at:"
+    echo "  Path: secret/data/langfuse/config"
+    echo ""
+    echo "The LLM Orchestration Service will now be able to:"
+    echo "  - Initialize Langfuse client automatically"
+    echo "  - Track LLM usage and costs"
+    echo "  - Monitor orchestration pipelines"
+    echo ""
+    echo "Next steps:"
+    echo "  1. Restart llm-orchestration-service container (if running)"
+    echo "  2. Check logs for 'Langfuse client initialized successfully'"
+    echo "========================================"
+else
+    echo "Warning: Secrets stored but verification failed"
+    echo "  The secrets may still be accessible, but verification could not confirm."
+fi
diff --git a/vault/README.md b/vault/README.md

Original file line number	Diff line number	Diff line change
`@@ -1884,7 +1884,7 @@ def _refine_user_prompt(`
`1884`	`1884`	`)`
`1885`	`1885`	`output_json = validated_output.model_dump()`
`1886`	`1886`	`logger.info(`
`1887`		`- f"Prompt refinement output: {json.dumps(output_json, indent=2)}"`
	`1887`	`+ f"Prompt refinement output: {json_module.dumps(output_json, indent=2)}"`
`1888`	`1888`	`)`
`1889`	`1889`
`1890`	`1890`	`logger.info("Prompt refinement completed successfully")`