Sync with Development #556
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.
New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:
/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions
* added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages
* Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity
* add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.