feat(studio): rework into pipelines/workflow #48
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Studio migration support:
- Add StudioSessionStatus and StudioToolStatus enums
- Add StudioSession, StudioToolCall, StudioOperation models with New/Update variants
- Add StudioSessionRepository, StudioToolCallRepository, StudioOperationRepository
- Add constraint types for all studio tables
- Add error handlers for studio constraint violations
Account endpoint:
- Add GET /accounts/{accountId}/ endpoint to retrieve account by ID
- Add accounts_share_workspace query to check workspace membership
- Users can only view accounts they share a workspace with
…ve websocket - Add version_number column to document_files for file versioning - Add version query methods: list_file_versions, find_latest_version, get_next_version_number - Rename studio to chat throughout codebase (migrations, models, handlers) - Add chat handler with SSE streaming support - Add FILE_SOURCE enum (uploaded, imported, generated) - Update REQUIRE_MODE to content types (document, image, spreadsheet, etc.) - Remove WebSocket handler and workspace event streams - Remove ws features from axum and aide - Add EMBEDDING_DIMENSIONS constant (1536) - Upgrade rig-core from 0.12 to 0.28 - Add platform vision documentation in /docs - Clean up unused constants
- Add new nvisy-runtime crate for runtime services - Implement archive extraction and handling - Add archive error handling to server - Update document file constraints and models - Add SSH access documentation for private dependencies - Update migrations for document handling
… backends - Rename nvisy-workflow crate to nvisy-runtime - Move RuntimeConfig and RuntimeService into nvisy-runtime - Delete nvisy-runtime archive functionality (was separate crate) - Update nvisy-opendal: remove fs/memory backends - Add cloud storage backends: azblob, gdrive, dropbox, onedrive - Apply derive improvements (thiserror, derive_more) - Add prelude module with doc(hidden) - Update sources/sinks to match opendal providers: - S3, Gcs, AzureBlob, GoogleDrive, Dropbox, OneDrive - Remove FileStorage and DirectoryWatch - Delete delete_multiple_files endpoint from files handler - Delete archive endpoints and ArchiveService from server - Add pipeline handler with CRUD operations - Add pipeline-specific permissions - Update file/annotation models and handlers
…figs - Create nvisy-vector crate with qdrant, milvus, pinecone, pgvector backends - Reorganize nvisy-opendal with folder-per-backend structure (s3, gcs, azblob, gdrive, dropbox, onedrive) - Remove all feature gates from nvisy-opendal and nvisy-vector - Remove prelude modules from nvisy-core, nvisy-opendal, nvisy-vector, nvisy-runtime - Milvus and Pinecone backends are stub implementations pending SDK API alignment
martsokha
changed the title
martsokha
added
docs
…line workers - Add nvisy-data foundational crate with DataInput, DataOutput, VectorOutput traits - Move nvisy-rig from crates/ to integrations/ - Update nvisy-vector and nvisy-opendal to use nvisy-data traits - Remove pipeline worker folder from nvisy-server (keep handlers/models) - Remove nvisy-rig dependency from nvisy-server
…terns Stream module: - Rename publisher.rs → stream_pub.rs, subscriber.rs → stream_sub.rs - Merge file_job.rs into event.rs - Create EventStream trait with FileStream and WebhookStream implementations - Make EventPublisher<T, S> and EventSubscriber<T, S> generic over stream type - Remove EventPriority enum and priority field from FileJob - Remove with_consumer_name and with_filter from EventSubscriber - Add WebhookStream for webhook delivery (24h TTL) Object module: - Create ObjectBucket trait with NAME, DESCRIPTION, MAX_SIZE constants - Create ObjectKey trait with PREFIX constant - Make ObjectStore<B, K> generic over bucket and key types - Consolidate FileKey and AccountKey into object_key.rs - Add convenience methods: file_store, intermediates_store, thumbnail_store, avatar_store KV module: - Create KvBucket trait with NAME, DESCRIPTION, TTL constants - Create KvKey trait for key types (SessionKey, TokenKey) - Make KvStore<K, V, B> generic over key, value, and bucket - Delete ApiTokenStore, ChatHistoryStore, CacheStore wrappers - Add with_ttl constructor for runtime TTL override NatsClient: - Group methods into impl blocks: core, kv, object, stream - Add typed convenience getters for all store types - Remove unused dependencies: async-stream, strum, bytes Webhook implementation: - Add WebhookWorker for consuming and delivering webhooks - Add WebhookEmitter service for publishing webhook events - Update nvisy-server to use new webhook infrastructure
…split - Add nvisy-dal crate with unified data abstraction layer - DataTypeId enum and AnyDataValue for runtime type dispatch - ProviderConfig for S3, GCS, Azure Blob, Postgres, MySQL - Core module with Context, InputStream, OutputStream, DataInput/DataOutput traits - Storage and vector providers (Qdrant, Pinecone, Milvus, pgvector) - Add provider module to nvisy-runtime/node - ProviderParams: non-sensitive params with credentials_id reference - ProviderCredentials: sensitive credentials stored per workspace - CredentialsRegistry: in-memory registry with JSON construction - Update InputNode and OutputNode to use ProviderParams - Add CredentialsRegistry and CredentialsNotFound error variants - Remove nvisy-opendal, nvisy-vector, nvisy-data crates (consolidated into nvisy-dal)
…ecret - Move node types (input, output, transformer, data, id) into graph module - Create provider module with backend/, inputs.rs, outputs.rs, registry.rs - Split provider files into backend/ subdirectory (s3, gcs, azblob, postgres, mysql, qdrant, pinecone, milvus, pgvector) - Add read_data!/write_data! macros for DRY helper functions - Update ExecutionContext to use Vec<AnyDataValue> for 1→N transformations - Add secret column to workspace_webhooks migration
…-gated Ollama - Add provider/embedding module with EmbeddingModel, EmbeddingCredentials, EmbeddingProvider - Add provider/completion module with CompletionModel, CompletionCredentials - Add provider/splitting module (moved from rag/splitter) with improved observability - Feature-gate Ollama support behind 'ollama' feature flag (default enabled) - Remove old config.rs and registry.rs (replaced by type-safe enums) - Simplify error.rs with EmbeddingError variant - Update rig-core to 0.29 - Store rig EmbeddingModel directly in EmbeddingProvider variants (except Ollama)
…oning - Add agent module with 5 specialized agents: - VisionAgent: VLM tasks (image description, OCR, object detection) - TableAgent: table processing and format conversion - TextAnalysisAgent: NER, keywords, classification, sentiment - TextGenerationAgent: summarization, titles, contextual chunking - StructuredOutputAgent: JSON conversion with schema validation - Add Agents struct for convenient access to all agents - Refactor CompletionProvider and EmbeddingProvider to use Arc for cheap cloning - Rename inner enums to CompletionService and EmbeddingService - Split completion provider.rs into provider.rs, response.rs, rig_impl.rs - Split embedding provider.rs into provider.rs, rig_impl.rs - Add strum derives (AsRefStr, Display, EnumString) to model enums - Add PromptError variant to Error enum - Refactor transform configs: - ChunkConfig: add contextual_chunking, rename overlap - PartitionConfig: add PartitionStrategy enum (Auto, Fast, Slow, Vlm) - EmbeddingConfig: add normalize field - Add EnrichConfig, ExtractConfig, DeriveConfig for LLM tasks
…ph module - Add AI provider backends (OpenAI, Anthropic, Cohere, Gemini, Perplexity) - Add IntoProvider trait for unified params + credentials -> provider pattern - Add AI credentials to ProviderCredentials enum - Update transform configs to use *ProviderParams with credentials_id - Add validate() credentials check against CredentialsRegistry - Implement IntoProvider for InputProviderParams and OutputProviderParams - Split workflow.rs into workflow/ module (definition, edge, metadata, node) - Move WorkflowGraph to graph/graph.rs, add WorkflowDefinition for serialization - Add agent tools module with 8 specialized tools - Add agent memory module
- Add definition/ module with serializable, frontend-friendly types: - WorkflowDefinition, NodeDef, InputDef, OutputDef, SwitchDef - CacheSlot for inter-node data passing - Edge, EdgeData, NodeId, WorkflowMetadata - Add compiled/ module with runtime execution types: - CompiledGraph, CompiledNode, CompiledInput, CompiledOutput - CompiledTransform with processor structs for each transform type - CompiledSwitch for runtime routing - InputStream, OutputStream, DataStream, DataSink - Add compiler.rs for WorkflowDefinition -> CompiledGraph compilation: - Cache slot resolution - Credential lookup and provider instantiation - Graph validation - Refactor provider backends to return providers directly: - Make IntoProvider trait async using async_trait - Remove intermediate Config types - Simplify flow: Params + Credentials -> Provider - Remove legacy graph modules: - graph/input/, graph/output/, graph/route/, graph/workflow/ - graph/core.rs (WorkflowGraph) - Legacy executor methods - Update engine executor to use only CompiledGraph BREAKING CHANGE: WorkflowGraph removed, use WorkflowDefinition + compiler
- Add Position field to Node definitions for visual editor support
- Create compiled/input folder with stream.rs for InputStream/DataStream
- Create compiled/output folder with stream.rs for OutputStream/DataSink
- Split compiled/transform.rs into separate processor files
- Add Process trait to compiled/transform for runtime processing
- Move graph/transform into graph/definition/transform
- Remove Transform trait from definitions (now pure data types)
- Simplify SwitchDef to single condition with match/else ports
- Add derive_builder to Edge, Node, and WorkflowMetadata
- Use Jiff Timestamp for FileDateCondition
Structure:
compiled/{input,output,transform}/ - runtime types with Process trait
definition/transform/ - serializable transform definitions
…separation - Move graph/definition to src/definition (top-level module) - Move graph/compiled to src/graph (renamed from compiled) - Move compiler.rs into engine folder (private module) - Engine.execute() now accepts WorkflowDefinition directly - Rename graph/graph.rs to compiled.rs to avoid module inception warning - Update all imports across nvisy-runtime and nvisy-server
…ation - Split route module into definition/route (cache, switch) and graph/route (evaluators) - Simplify switch types to FileCategory and Language conditions with bool output - Add dedicated evaluators: FileCategoryEvaluator, LanguageEvaluator - Create engine::Context type with From impls for nvisy_dal::core::Context - Move EdgeData to graph/edge.rs (runtime type, not definition) - Inline NodeKind into Node struct, remove generic wrapper - Add Position type in definition/util module - Move Workflow directly into definition/mod.rs - Remove node types section from README - Delete unused stream module files
… providers - Refactor DataInput/DataOutput traits to use associated types for Item and Context - Split context.rs into separate files: ObjectContext, RelationalContext, VectorContext - Split stream.rs into input_stream.rs and output_stream.rs - Reorganize providers: keep struct in mod.rs, move DataInput impl to input.rs, DataOutput impl to output.rs - Vector DB providers (qdrant, pinecone, milvus, pgvector) only have output.rs (write-only) - Remove unused ProviderConfig enum - Update error.rs to use thiserror and add BoxError type alias - Update nvisy-runtime to use new trait structure - Add strum derive for ProviderCredentials.kind() - Clean up provider/mod.rs re-exports
…beddingProvider - Move IntoProvider trait to nvisy-core as shared abstraction - Implement From<nvisy_rig::Error> for nvisy_core::Error - Refactor CompletionProvider to use IntoProvider::create(params, credentials) - Refactor EmbeddingProvider to use IntoProvider::create(params, credentials) - Update runtime backend providers to use new create pattern - Provider methods now return nvisy_core::Result for consistency
… module - Add ApiKeyCredentials and OllamaCredentials shared types in nvisy-rig - Update CompletionCredentials and EmbeddingCredentials to use shared types - Remove IntoAiProvider trait and backend module from nvisy-runtime - Add strum derives for kind() method on nvisy_core::Error - Implement IntoProvider trait in nvisy-core for provider abstraction - Add From<nvisy_dal::Error> and From<nvisy_rig::Error> for nvisy_core::Error
- Add JsonSchemaTool<T> using schemars for schema generation and jsonschema for validation - Add JsonResponse for parsing JSON from LLM responses (handles markdown code blocks) - Create concrete schema types for agents: ColumnDescription, TextAnalysisOutput, StructuredOutput - Add with_tools flag to all agents for optional tool enablement - Add memory module with ChatHistory (compaction strategies) and WorkingMemory - Rename tools/ to tool/ and make module private - Update runtime processors to use specific agents instead of Agents bundle - Box large processor variants (EnrichProcessor, ExtractProcessor) in CompiledTransform - Remove unused chat, service, session, and tool modules
- Update all license mentions from MIT to Apache 2.0 - Add changelog/license/support sections to all crate READMEs - Add build badge to crates/README.md - Standardize Cargo.toml structure: name, description, readme, keywords, categories first - Add keywords and categories to all crate Cargo.toml files
## GitHub Actions Fixes - Fix broken intra-doc links in nvisy-runtime documentation - Update documentation to reference public Engine API instead of private compiler - All CI checks now pass (format, check, clippy, tests, docs) ## Security Improvements - Remove Milvus provider entirely - eliminates 4+ critical security vulnerabilities - Remove milvus-sdk-rust dependency and all related security advisories: - RUSTSEC-2025-0009 (ring AES panic with overflow checking) - RUSTSEC-2025-0010 (unmaintained ring versions) - RUSTSEC-2023-0071 (rsa Marvin Attack timing vulnerability) - RUSTSEC-2024-0336 (rustls infinite loop vulnerability) - Update deny.toml configuration for remaining dependencies - cargo deny check now passes with only duplicate dependency warnings ## Code Quality - Fix clippy warning about large enum variants using Box<> for providers - Keep Qdrant, Pinecone, and pgvector as vector database alternatives - Maintain all functionality while removing security risks ## Documentation Cleanup - Abstract DAL README to remove specific database names and code examples - Make provider documentation more generic and focused on concepts - Remove implementation details that expose internal architecture All 198 tests pass, documentation builds cleanly, and security posture significantly improved.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.