Development to feature/code-highlightings #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Support for custom environment (microsoft#235) (microsoft#240) Co-authored-by: adu103 <adu103@users.noreply.github.com> Co-authored-by: Alex Unger (Dev Advocate) <alexunger@microsoft.com> * Fix code snippet readability in Dark Mode (microsoft#239) * Initial plan for issue * Add dark mode styling for code blocks Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * Fix dark mode file link contrast in chat messages (microsoft#241) * Initial plan for issue * Fix dark mode file link contrast by using white text on green background Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * update version to 0.215.001 * Admin Setting - Use Local File for Document Intelligence Testing (microsoft#245) * Initial plan for issue * Update Document Intelligence test to use local file Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * Redis (microsoft#251) * adding ui to admin page for redis settings * removing APIM from redis * updates so data being saved in cosmos correct and test button working * Session data is correctly being stored in redis * update test redis button * Have the extra fields for redis auth type now working correctly * finished support for managed identities for redis cache * update readme for redis * fixed issue with redis not working with scaled app and managed indentities --------- Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> * Changing Base Image to reduce CVEs (microsoft#266) * Update Dockerfile * Update docker_image_publish_dev.yml * Update Dockerfile to harden * Update Dockerfile Changed back to 3.12-slim * Update Dockerfile to remove libaom * Update Dockerfile Changed base to chainguard * Update docker_image_publish_dev.yml Changed image branches to trigger * Update Dockerfile Changed base to latest * Update Dockerfile removing the installs * Update Dockerfile adding latest-dev layer * Update Dockerfile remove flask * Update Dockerfile Changing EntryPoint * Update Dockerfile Using venv * Update app.py Updating app.run to use 0.0.0.0 * Update Dockerfile Giving User write access to image dir * Update Dockerfile Fixing to support chown without distro * Update Dockerfile update EXPOSE comment * add appinsights logger and function * fix improperly nested function * rmv duplicate function * add appinsights lib * fix version decl * improved logger options * Update chats.css (microsoft#272) Fix word wrap to move word to next line in chat * Infra deployer gunger (microsoft#232) * deployers update cleaning up code files for commit. * add deployer options readme.md add deployer options readme.md * Notes cleanup Notes cleanup * One Touch Deploy * update to one touch deploy links update to one touch deploy links * OneTouch updates OneTouch updates * One Touch Deploy changes One Touch Deploy changes * Update README.md * update to json template update to json template * Update README.md * Update README.md * Update README.md * Update README.md * Update OneTouchDeploy.md * Update OneTouchDeploy.md * Changed name of one touch to one click Changed name of one touch to one click * update to bicep * update to registry setting that differs by platform * update to variables to make it compatible with Azure Commercial update to variables to make it compatible with Azure Commercial * Update README.md * document bulk loader first checkin * database seeder first checkin * db seeder updates * updates to external api * bulk uploader changes bulk uploader changes * bulk loader related cleanup bulk loader related cleanup * bulk document loader changes bulk document loader changes * bulk loader updates bulk loader updates * Update main.py * Update main.py Bulk loader changes * Update route_external_group_documents.py External api changes * database seeder changes database seeder changes * database seeder updates database seeder updates * database seeder updates database seeder updates * Update admin_settings.json --------- Co-authored-by: Greg Unger <117244765+gregunger_microsoft@users.noreply.github.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> * Support using Azure File Share for temp files (microsoft#264) Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> * Update test_document.pdf * upate pipeline * init semantic kernel commit --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: adu103 <adu103@users.noreply.github.com> Co-authored-by: Alex Unger (Dev Advocate) <alexunger@microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> Co-authored-by: eldong <eldon@sixarrows.net> Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> Co-authored-by: Aaron Barth <aaronba@users.noreply.github.com> Co-authored-by: Greg <78557883+gregunger-microsoft@users.noreply.github.com> Co-authored-by: Greg Unger <117244765+gregunger_microsoft@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com>
Updated typo for branch so we can test the build
…into Development
* fix saving switching to kwargs * fix the description on plugin and rmv extra field on save * disable kernel redis saving * add allowed keys * upd defaults for greenfield deployments * add enhancements to fact memory * upd get_fact and disable optional filters * Updated version --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* fix global apim settings issues * add explicit per agent/kernel apiversion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* Support for custom environment (microsoft#235) (microsoft#240) Co-authored-by: adu103 <adu103@users.noreply.github.com> Co-authored-by: Alex Unger (Dev Advocate) <alexunger@microsoft.com> * Fix code snippet readability in Dark Mode (microsoft#239) * Initial plan for issue * Add dark mode styling for code blocks Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * Fix dark mode file link contrast in chat messages (microsoft#241) * Initial plan for issue * Fix dark mode file link contrast by using white text on green background Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * update version to 0.215.001 * Admin Setting - Use Local File for Document Intelligence Testing (microsoft#245) * Initial plan for issue * Update Document Intelligence test to use local file Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> * Redis (microsoft#251) * adding ui to admin page for redis settings * removing APIM from redis * updates so data being saved in cosmos correct and test button working * Session data is correctly being stored in redis * update test redis button * Have the extra fields for redis auth type now working correctly * finished support for managed identities for redis cache * update readme for redis * fixed issue with redis not working with scaled app and managed indentities --------- Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> * Changing Base Image to reduce CVEs (microsoft#266) * Update Dockerfile * Update docker_image_publish_dev.yml * Update Dockerfile to harden * Update Dockerfile Changed back to 3.12-slim * Update Dockerfile to remove libaom * Update Dockerfile Changed base to chainguard * Update docker_image_publish_dev.yml Changed image branches to trigger * Update Dockerfile Changed base to latest * Update Dockerfile removing the installs * Update Dockerfile adding latest-dev layer * Update Dockerfile remove flask * Update Dockerfile Changing EntryPoint * Update Dockerfile Using venv * Update app.py Updating app.run to use 0.0.0.0 * Update Dockerfile Giving User write access to image dir * Update Dockerfile Fixing to support chown without distro * Update Dockerfile update EXPOSE comment * add appinsights logger and function * fix improperly nested function * rmv duplicate function * add appinsights lib * fix version decl * improved logger options * Update chats.css (microsoft#272) Fix word wrap to move word to next line in chat * Infra deployer gunger (microsoft#232) * deployers update cleaning up code files for commit. * add deployer options readme.md add deployer options readme.md * Notes cleanup Notes cleanup * One Touch Deploy * update to one touch deploy links update to one touch deploy links * OneTouch updates OneTouch updates * One Touch Deploy changes One Touch Deploy changes * Update README.md * update to json template update to json template * Update README.md * Update README.md * Update README.md * Update README.md * Update OneTouchDeploy.md * Update OneTouchDeploy.md * Changed name of one touch to one click Changed name of one touch to one click * update to bicep * update to registry setting that differs by platform * update to variables to make it compatible with Azure Commercial update to variables to make it compatible with Azure Commercial * Update README.md * document bulk loader first checkin * database seeder first checkin * db seeder updates * updates to external api * bulk uploader changes bulk uploader changes * bulk loader related cleanup bulk loader related cleanup * bulk document loader changes bulk document loader changes * bulk loader updates bulk loader updates * Update main.py * Update main.py Bulk loader changes * Update route_external_group_documents.py External api changes * database seeder changes database seeder changes * database seeder updates database seeder updates * database seeder updates database seeder updates * Update admin_settings.json --------- Co-authored-by: Greg Unger <117244765+gregunger_microsoft@users.noreply.github.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> * Support using Azure File Share for temp files (microsoft#264) Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> * Update test_document.pdf * upate pipeline * init semantic kernel commit * bug fix for fact memory * fix for sk * disable kernel redis saving * trying to trigger a deploy that works (old code still showing after several builds) * add comments for partial deployment * rmv app.run settings * add back * adding imports back * upd get facts * add log analytics plugin base * init of combined plugin logic * add docs for agents/plugins * add always populate addfields and metadata * fix bug with walkthrough * fix copilot found bugs * fix bug in plugin updater * add LAW default global agent * fix LAW global setting * fix for admin settings --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: adu103 <adu103@users.noreply.github.com> Co-authored-by: Alex Unger (Dev Advocate) <alexunger@microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: paullizer <34814295+paullizer@users.noreply.github.com> Co-authored-by: eldong <eldon@sixarrows.net> Co-authored-by: Eldon Gormsen <Eldon.Gormsen@microsoft.com> Co-authored-by: Aaron Barth <aaronba@users.noreply.github.com> Co-authored-by: Greg <78557883+gregunger-microsoft@users.noreply.github.com> Co-authored-by: Greg Unger <117244765+gregunger_microsoft@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com>
Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* Update config.py to add Login Redirect URL env setting * Update functions_authentication.py Adding Auth code for redirect * Update docker_image_publish_dev.yml * Adding another environment variable for HOME_REDIRECT_URL * Adding logging for HOME_REDIRECT_URL logic and LOGIN_REDIRECT_URL logic * Update docker_image_publish_dev.yml Updating GH Actions trigger branch * Update config.py to handle the Non- Front Door condition better * Update docker_image_publish_dev.yml * Adding support for /logout * Update docker_image_publish_dev.yml Quick update for the merge * Initial plan * Add admin-configurable HOME_REDIRECT_URL and LOGIN_REDIRECT_URL settings Co-authored-by: aaronba <5930919+aaronba@users.noreply.github.com> * Move Authentication & Redirect Settings to Scale tab as requested Co-authored-by: aaronba <5930919+aaronba@users.noreply.github.com> * Update docker_image_publish_dev.yml Updating GH Actions trigger branch * Update docker_image_publish_dev.yml Updating GH Actions trigger branch to Development --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: aaronba <5930919+aaronba@users.noreply.github.com>
* fix issue with group user search * add test claims * init agent chat selector * adds simple agent mode --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com>
* fixed fetch model using regex and pandas instead of pd * Added public workspace * Update config.py * Update * Pub workspace detail row, edit metadata, and extract metadata * public workspace * public directory * Tied public workspace to chat * Save public workspace visibility to user settings * public workspace chat sends to chat page * create list of visible public workspaces * Updated public directory UI * correct agents_common.js location
* fix issue with group user search * add test claims * init agent chat selector * adds simple agent mode * fix workspace to use new modal + update modal plugins to select * revert to app auth * init admin fine control over workspaces * Update application/single_app/static/js/admin/admin_settings.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py fix incorrect logic in validation Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fixed public workspace file processing and updated index jsons * add extract metadata to process_ functions * removed unused functions that live in functions_documents.py * spacing * updated search_groups function description * added search support for public workspace * cleaned up spacing in function_settings.py * added support for public workspaces * added support for public workspace * added support for file sharing * added file sharing to admin settings * file sharing functional * Update config.py * enable file sharing * Shared files require approval by receiver
* fix issue with group user search * add test claims * init agent chat selector * adds simple agent mode * fix workspace to use new modal + update modal plugins to select * revert to app auth * init admin fine control over workspaces * Update application/single_app/static/js/admin/admin_settings.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py fix incorrect logic in validation Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing decorator * upd to unified logic and fix global agent bug * fix for errors workspace agents * upd namespace for module * add modal error clear * add logging to loader * rmv forced deployment * add logging for loader * fix loader logging * change alert to toast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing import calls --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…into Development
* fix issue with group user search * add test claims * init agent chat selector * adds simple agent mode * fix workspace to use new modal + update modal plugins to select * revert to app auth * init admin fine control over workspaces * Update application/single_app/static/js/admin/admin_settings.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py fix incorrect logic in validation Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing decorator * upd to unified logic and fix global agent bug * fix for errors workspace agents * upd namespace for module * add modal error clear * add logging to loader * rmv forced deployment * add logging for loader * fix loader logging * change alert to toast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing import calls * rmc extra "" and fix unintended call into obj * upd alert in showAgentModalError * fix gpt deployment name processing * rmv deprecated plugins_to_load * fix showtoast --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix issue with group user search * add test claims * init agent chat selector * adds simple agent mode * fix workspace to use new modal + update modal plugins to select * revert to app auth * init admin fine control over workspaces * Update application/single_app/static/js/admin/admin_settings.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/route_backend_plugins.py fix incorrect logic in validation Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing decorator * upd to unified logic and fix global agent bug * fix for errors workspace agents * upd namespace for module * add modal error clear * add logging to loader * rmv forced deployment * add logging for loader * fix loader logging * change alert to toast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add missing import calls * rmc extra "" and fix unintended call into obj * upd alert in showAgentModalError * fix gpt deployment name processing * rmv deprecated plugins_to_load * fix showtoast * fix for loader * fix resolver --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
… Development branch (microsoft#361)
) * fixed public workspace file processing and updated index jsons * add extract metadata to process_ functions * removed unused functions that live in functions_documents.py * spacing * updated search_groups function description * added search support for public workspace * cleaned up spacing in function_settings.py * added support for public workspaces * added support for public workspace * added support for file sharing * added file sharing to admin settings * file sharing functional * Update config.py * enable file sharing * Shared files require approval by receiver * Added ability to deny file share at the approval/deny pop up modal * added group file share * Adding group search for files that are shared * fixed table row alignment bug * Added Go to Group Workspace button in Manage Group page * fixed drop down double chevron bug * Update config.py * public workspaces functional in chats * updated chat api to support modifying the top_n value, defaults to 12 goal is more so to support agents calling the chat model and providing more documents. * replaced manually upload, now area with auto upload * Updated public workspace upload workflow * cleaned up the prompt tab in the workspaces * Simplified UI for action buttons * Support public workspace upload * side nav bar * top and side nav bar * Nav bar work * side bar nav ui issues * side bar and top nav updates * better support for mobile * Updated my account menu * side nav bar supports conversations * side bar nav with conversation management * Updated chat page input and buttons * Added Left Nav to Appearence tab in Admin Settings * Updated Message Input UI, Logo * Light mode and dark mode logo support * Conversations working in both top nav and side bar
🔗 Feature: External Links Navigation System 📋 Overview Added a comprehensive External Links management system to allow administrators to add custom navigation links to external resources. This feature provides a flexible way to integrate external tools, policies, documentation, and other resources directly into the SimpleChat navigation interface. ✨ Key Features Admin Configuration ✅ Enable/Disable Toggle: Administrators can enable or disable external links in navigation ✅ Custom Menu Name: Configurable menu title (defaults to "External Links") ✅ Force Menu Display: Option to always show as dropdown vs. smart display logic ✅ Dynamic Link Management: Add, edit, and delete external links with validation Smart Navigation Display 🎯 Adaptive UI Logic: 1-2 links + Force Menu OFF: Shows as individual top-level navigation items 3+ links OR Force Menu ON: Shows as dropdown menu with custom name 🔄 Dual Navigation Support: Works in both top navigation and sidebar layouts 🛡️ Security: All external links open in new tabs with rel="noopener noreferrer"
* profile image * Improve profile image loading speed.
* Conversation metadata * Updated with Human Readable format fields
paullizer
pushed a commit
that referenced
this pull request
Dec 19, 2025
* add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 14, 2026
* creating workflows * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 14, 2026
* creating workflows * fraud analyssis * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print * Updated README * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 14, 2026
* creating workflows * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print * Updated README * Update README.md * accepted changes --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 14, 2026
* creating workflows * fraud analyssis * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print * Updated README * Update README.md * accepted changes * removed files * GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app * Upated to v0.235.002 --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 14, 2026
* creating workflows * fraud analyssis * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print * Updated README * Update README.md * accepted changes * removed files * GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app * Upated to v0.235.002 * removed debug test file * Updated to v0.235.003 --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Xeelee33
added a commit
that referenced
this pull request
Jan 15, 2026
* creating workflows * fraud analyssis * support agents * update * fix * updated demo * Swagger lite (microsoft#469) * Development (microsoft#467) * upgrade to v0.229.060 (microsoft#459) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * docs demos for public workspaces * V0.229 bug fixes (v0.229.019) (microsoft#448) * Development (microsoft#445) * Update release notes to show support for GPT-5 * Documented support for gpt-image-1 * Update config.py * remove documentation folder * Documentation and message table support (microsoft#444) * Develop demo docs and import markdown table support * fixed enhanced citations for groups and public workspaces * Updated to support showing public workspaces in scope * Update config.py * fix docs * Updated RELEASE_NOTES * video indexer config details, doc intel test button fix, move multimedia configs to search and extract * improved header security * updated versions * moved * Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md * added pdfs * v0.229.019 bug fixes upgrade to v0.229.058 (microsoft#452) * all urls in chat open in new tabs * consolidated admin settings for improved navigation * added left hand nav admin settings menus * added left hand menu options for workspaces * Added debug logging to video indexer processes * readme and functional test * Workspace Scope in Chat affects Prompts * Create WORKSPACE_SCOPE_PROMPTS_FIX.md * time based turn off for debug and file process logging * improve saving in admin settings * update to v0.229.058 * Update RELEASE_NOTES.md * Update RELEASE_NOTES.md * Popup modal for Health Check config * Added Health Check config guide * Chat page top nav bug (microsoft#458) * initial fix * fixed top nav chat up bug * notes for v0.229.060 * file location fix * Update config.py * Update RELEASE_NOTES.md * moved to correct location * Fixed enhanced citations CSP bug Simple Chat implemented improved security which negatively impacted enhanced citations. * Updated release notes * updated version and tests * swagger support for all endpoints and added swagger search * added wide screen support for chats when collapsing side bar * v0.230.001 features * adding support for xlsm, Macro Excel files. * moved into features * initial * added readme * removed html code * Update config.py (microsoft#477) Updated else if for AUTHORITY * Initial Setup for Pages documentation (microsoft#479) * setup folders and base files * setting up files * architecture diagrams * updated to libdoc * libdoc updates * updating side bar * removed loops * editing side bar * Created Simple Chat Jekyll theme * Update config.py (microsoft#477) (microsoft#478) Updated else if for AUTHORITY Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * Updating architectures * Update README.md --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> * initial * added to base * adding real data endpoints * Update route_backend_control_center.py * added individual charts * fix for bug 485 * added document metrics * added links to control center * debug * added date * fixed bugs due to branch descrepancies * added Azure SQL Driver Docker File * added documentation for docker_fileSession updates * Redis Managed Identity Azure Government Support Changes * Stop tracking ignored folders * updated gitignore * added sort by to table for user management * storage account size processing * Front end now shows storage account sizing * export user management list to csv * adding group management * fixing swagger generation * fix * Added inline dynamic property generation * added YAML support * Improved muiltform vs app/json detection * added Control Center Admin role ControlCenterAdmin * ai search sizing is working for groups * group refresh fixed * added group data fix * group table refresh * updated export to include group docs * adding public workspace management * removed sample data and consolidated row generators * Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert. * removed workflow, will work on that in different branch * Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization. * added debug logging * setup cache feature and ttl time to admin app settings * removed cosmos level ttl * Keyvault for secrets (microsoft#492) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/remove abp for pr (microsoft#510) * add crude keyvault base impl * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * add ui trigger word and get agent helper * adds safe calls for akv functions * add keyvault to global actions loading * rmv secret leak via logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add security tab UI + key vault UI * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/group agents actions (microsoft#521) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add group agents/actions * add branch for testing/rmv old branch * bug fixes, group agent modifications, rmv client validation * rmv ajv * upd from copilot --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Add cosmos activity logs container configuration * incorporate branch updates Add 372 fix 489 * Support deployment via AZD UP (microsoft#530) * Update devcontainer configuration for support of AZD * Move to module based bicep files * Add Azure deployment configuration and update Bicep modules for service outputs * Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs. * Add deployment script for creating and storing Azure AD client secret in Key Vault * Update Azure Dev CLI feature version to latest in devcontainer configuration * Remove deprecated Bicep files and parameter configurations for cleaner deployment structure * Refactor Bicep modules for improved diagnostics and role assignments - Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter. - Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings. - Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret. - Modified appServicePlan.bicep to conditionally import diagnostic settings. - Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group. - Updated azureContainerRegistry.bicep to conditionally import diagnostic settings. - Enhanced contentSafety.bicep with conditional diagnostic settings import. - Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity. - Refactored documentIntelligence.bicep to conditionally import diagnostic settings. - Enhanced enterpriseApplication.bicep by adding additional required resource access scopes. - Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters. - Refactored openAI.bicep to conditionally import diagnostic settings. - Enhanced redisCache.bicep with conditional diagnostic settings import. - Updated search.bicep to conditionally import diagnostic settings. - Refactored speechService.bicep to conditionally import diagnostic settings. - Enhanced storageAccount.bicep with conditional diagnostic settings import. - Added main.parameters.json for parameter management. - Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments. * Add custom subdomain names for document intelligence, OpenAI, and speech services * Fix casing for hostingMode property in search service configuration * Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers * Add enterprise app permissions module for resource access management * Fixed ExternalApi configuration to valid guid and set value to a unique name * Add Init Script to Configure Entra Application * Fix spelling error * fix failure in hostingMode value * configure managed identity for contentSafety * update readme to support new AZD deployment solution * Video Indexer, Multi-Modal Enhancements, Scope Bug ## PR Summary: Video Indexer Multi-Modal Enhancements ### Overview This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**. ### 🎯 Key Features #### 1. **Multi-Modal Vision Analysis for Images** - Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models - Extracts comprehensive image insights including: - AI-generated descriptions - Object detection - Text extraction from images (OCR) - Detailed visual analysis - New admin setting: `enable_multimodal_vision` to control feature availability - Vision analysis results stored in document metadata and included in AI Search indexing - Connection testing endpoint added for vision model validation #### 2. **Enhanced Document Metadata Citations** - Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis - New citation types displayed with distinct visual indicators: - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata" - **Abstract**: Document summaries included as contextual citations - **Vision Analysis**: AI-generated image insights labeled as "AI Vision" - Metadata content passed to AI models as additional context for more informed responses - Special modal view for metadata citations (separate from standard document citations) #### 3. **Image Message UI Improvements** - Enhanced display for user-uploaded images vs AI-generated images - Added "View Text" button for uploaded images with extracted content or vision analysis - Collapsible info sections showing: - Extracted OCR text from Document Intelligence - AI Vision Analysis results - Proper avatar distinction between uploaded and generated images - Improved metadata tracking with `is_user_upload` flag #### 4. **Video Indexer Configuration Updates** - **BREAKING CHANGE**: Removed API key authentication support - Now exclusively uses **Managed Identity authentication** for Video Indexer - Updated admin UI documentation to guide managed identity setup: - Enable system-assigned managed identity on App Service - Assign "Video Indexer Restricted Viewer" role - Configure required ARM settings (subscription ID, resource group, account name) - Improved validation for required Video Indexer settings - Enhanced error messaging for missing configuration #### 5. **Search Scope Improvements** - Fixed search behavior when `document_scope='all'` to properly include group documents - Added `active_group_id` to search context when document scope is 'all' and groups are enabled - Conditional group index searching - only queries group index when `active_group_id` is present - Prevents unnecessary searches and potential errors when groups aren't in use #### 6. **Image Context in Conversation History** - Enhanced conversation history to include rich image context for AI models - Extracts and includes: - OCR text from Document Intelligence (up to max content length) - AI Vision analysis (description, objects, text) - Structured prompt formatting for multimodal understanding - **Important**: Base64 image data excluded from conversation history to prevent token overflow - Only metadata and extracted insights passed to models for efficient token usage ### 🔧 Technical Improvements #### Backend Changes - **route_backend_chats.py**: - Added metadata citation extraction logic (~150 lines) - Enhanced conversation history building for image uploads - Improved search argument handling for group contexts - **functions_documents.py**: - New `analyze_image_with_vision_model()` function for AI vision analysis - Enhanced `get_document_metadata_for_citations()` integration - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing - Removed redundant blob storage for vision JSON (stored in document metadata) - **route_backend_settings.py**: - New `_test_multimodal_vision_connection()` endpoint for testing vision models - Supports both APIM and direct Azure OpenAI endpoints - Test uses 1x1 pixel sample image for validation - **functions_search.py**: - Added conditional logic for group search execution - Prevents empty `active_group_id` from causing search errors #### Frontend Changes - **chat-messages.js** (~275 lines changed): - Enhanced `appendMessage()` to handle uploaded image metadata - New `toggleImageInfo()` functionality for expandable image details - Improved citation rendering with metadata type indicators - Debug logging for image message processing - **chat-citations.js** (~70 lines added): - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis - Enhanced citation click handling to detect metadata citations - Separate modal styling and behavior for metadata vs document citations - **admin_settings.html**: - Complete redesign of Video Indexer configuration section - Removed all API key references - Added managed identity setup instructions with step-by-step guidance - Updated configuration display to show resource group and subscription ID - **_video_indexer_info.html**: - Updated modal content to clarify managed identity requirement - Added warning banner about authentication type - Enhanced configuration display with ARM resource details ### 📊 Files Changed - **16 files** modified - **+1,063 insertions**, **-412 deletions** - Net change: **+651 lines** ### 🧪 Testing Considerations - Test multi-modal vision analysis with various image types - Validate metadata citations appear correctly in chat responses - Verify Video Indexer works with managed identity authentication - Test search scope behavior with and without groups enabled - Validate image upload UI shows extracted text and vision analysis - Confirm conversation history properly handles image context without token overflow ### 🔐 Security & Performance - Managed identity authentication improves security posture (no stored API keys) - Image base64 data excluded from conversation history prevents token exhaustion - Metadata citations add minimal overhead while providing rich context - Vision analysis runs efficiently during document processing pipeline ### 📝 Configuration Required Admins must configure: 1. Enable `enable_multimodal_vision` in admin settings 2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`) 3. For Video Indexer: Configure managed identity and ARM resource details 4. Enable `enable_extract_meta_data` to surface metadata citations --- This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices. * Conversation Management Features (microsoft#532) New Features 1. Pin Conversations Users can pin important conversations to keep them at the top of the list Pinned conversations display a pin icon (📌) in the conversation header and details modal Pin icon appears before the conversation title Bulk pin/unpin operations available in multi-select mode Pinned conversations always appear first, sorted by most recent activity 2. Hide Conversations Users can hide conversations to declutter their workspace without deleting them Hidden conversations display an eye-slash icon (👁️🗨️) in the conversation header and details modal Eye-slash icon appears next to the pin icon (if pinned) Bulk hide/unhide operations available in multi-select mode Toggle visibility of hidden conversations using the eye icon in the sidebar 3. Two-Tier Conversation Search Quick Search (Sidebar) Instant title-based filtering of conversations Search icon in sidebar activates inline search input Real-time filtering as you type Clear button to reset search Expand button to open advanced search modal Advanced Search (Modal) Full-text search across all message content Multiple filter options: Date range (from/to) Chat type (personal/group/public) Classifications (multi-select) Has uploaded files Has generated images Pagination (20 results per page) Message snippets with highlighted search terms (50 chars before/after match) Click to navigate directly to specific messages Search history tracking (last 20 searches) Clickable search history to repeat searches 4. Message Highlighting & Navigation Search results highlight matched text in yellow (amber in dark mode) Smooth scroll animation to navigate to specific messages Pulse animation draws attention to the target message Highlights persist for 30 seconds before auto-clearing Works across conversation switches 5. Multi-Select Mode Select multiple conversations for bulk operations Visual checkboxes appear when entering selection mode Bulk actions available: Pin/unpin selected conversations Hide/unhide selected conversations Delete selected conversations Selection mode accessible from conversation dropdown menu Auto-exit after 30 seconds of inactivity 6. Enhanced Conversation Details Modal Displays pin icon if conversation is pinned Displays eye-slash icon if conversation is hidden Shows both icons at the top of the modal (next to title) Status section shows visual badges for pinned/hidden state Comprehensive metadata display Technical Implementation Frontend Changes chat-conversations.js: Core conversation management, quick search, pin/hide functionality chat-search-modal.js (NEW): Advanced search modal implementation chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling chat-messages.js: Message highlighting, smooth scroll, search highlight persistence chat-conversation-details.js: Updated to show pin/hidden icons in modal chats.css: Styles for search highlights and message pulse animations HTML Templates: Added search modal, updated navigation icons Backend Changes route_backend_conversations.py: /api/search_conversations - Full-text search with filters and pagination /api/conversations/classifications - Get unique classification values /api/user-settings/search-history - GET/POST/DELETE endpoints for search history /api/conversations/{id}/pin - Toggle pin status /api/conversations/{id}/hide - Toggle hide status Bulk operations for pin/hide/delete functions_settings.py: Search history management functions * Message management (microsoft#553) * added message masking mask selected content of message or an entire message * fixed citation border * enabled streaming * image gen with streaming * added reasoning support * added reasoning to agents * agent support * fixed key bug * disable group create and fixed model fetch * updated config * fixed support for workspace search for streaming * fix bug with sidebar update * fixed gpt-5 vision processing bug * metadata works with all messages now * fixed debug_print bug * added reasoning effort to agents and fixed agent validation * fixed file metadata loading bug * fixed llm streaming when working with group workspace data * fixed cosmos container config error * added delete message and fixed message threading * retry bug fixes * fixed message threading order * moved message buttons to menu * fixed bug for conversation history that included inactive threads * added css styling for urls for dark mode * fixed bug with newly created messages not showing metadata or deleting * improved search times by 100x * added token collect to messages supports models and agents * added streaming for agents along with token collection * added embedding token tracking * added document creation/deletion and token tracking to activity log * adding conversations to activity logs * added activity log viewer with filters, search, and export * added support for agents in edit and retry messages * Configure Application from AZD Up command (microsoft#548) * Add Cosmos DB post-configuration script and update requirements - Initial POC * post deploy configure services in cosmosdb * refactor to prevent post deploy configuration + begin support of key based auth. * Add additional parameter validation for creating entra app * Refactor Bicep modules for improved authentication and key management - Added keyVault-Secrets.bicep module for storing secrets in Key Vault. - Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments. - Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically. - Added setPermissions.bicep for managing role assignments for various resources. - Updated postconfig.py to reflect changes in environment variable handling for authentication type. * Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services * initial support for VideoIndexer service * Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations * move from using chainguard-dev builder image to python slim image. * Updates to support post deployment app config * Add post-deployment permissions script for CosmosDB and update authentication type handling * fix typo in enhanced citation deployment config * Refactor Dockerfile to use Python 3.13-slim and streamline build process * restart web application after deployment settings applied * remove setting for disableLocalAuth * update to latest version of bicep deployment * remove dead code * code cleanup / formatting * removed unnecessary content from readme.md * fix token scope for commericial search service * set permission correctly for lookup of openAI models * fixes required to configure search with managed identity * Adds Azure Billing Plugin in Community Customizations (microsoft#546) * add crude keyvault base impl * upd actions for MAG * add settings to fix * upd secret naming convention * upd auth types to include conn string/basic(un/pw) * fix method name * add get agent helper * add ui trigger word and get agent helper * upd function imports * upd agents call * add desc of plugins * fix for admin modal loading * upd default agent handling * rmv unneeded file * rmv extra imp statements * add new cosmos container script * upd instructions for consistency of code * adds safe calls for akv functions * adds akv to personal agents * fix for user agents boot issue * fix global set * upd azure function plugin to super init * upd to clean imports * add keyvault to global actions loading * add plugin loading docs * rmv secret leak via logging * rmv displaying of token in logs * fix not loading global actions for personal agents * rmv unsupported characters from logging * fix chat links in dark mode * chg order of css for links in dark mode * fix chat color * add default plugin print logging * rmv default check for nonsql plugins * upd requirements * add keyvault and dynamic addsetting ui * fix for agents/plugins with invalid akv chars * add imp to appins logging * add security tab UI + key vault UI * add keyvault settings * fix for copilot findings. * fix for resaving plugin without changing secret * init azure billing plugin * add app settings cache * upd to azure billing plugin * upd to msgraph plugin * init community customizations * add module * add key vault config modal * add logging and functions to math * rmv extra telemetry, add appcache * upd billing plugin * add/upd key vault, admin settings, agents, max tokens * Remove abp for pr * disable static logging for development * rmv dup import * add note on pass * added notes * rmv dup decl * add semicolon * rmv unused variable add agent name to log * add actions migration back in * add notes and copilot fixes * add abp back in * upd abp/seperate graph from query * rmv missed merge lines * fix for AL * upd for consistency testing * upd abp to community * fix copilot findings #1 * fix plotting conflict * fix exception handling * fix static max function invokes * rmv unneeded decl * rmv unneeded imports * fix grouping dimensions * fix abp copilot suggestions #2 * simplify methods for message reload * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots * upd pipeline to include branch name in image * add abp to deploy * upd instructions name/rmv abp from deploy * fix pipeline * mov back to Comm Cust for main inclusion --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Security/container build (microsoft#549) * upd dockerfile to google distroless * add pipelines * add modifications to container * upd to build * add missing arg * add arg for major/minor/patch python version * upd python paths and pip install * add perms to /app for user * chg back to root * rmv python3 * rmv not built python * add shared * add path and home * upd for stdlib paths * fix user input filesystem path vulns * fix to consecutive dots --------- Co-authored-by: Bionic711 <nadoyle@microsoft.com> * Feature/speech managed identity (microsoft#543) * Bugfix - deleted duplicate enable_external_healthcheck entry * Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs. * Update application/single_app/templates/admin_settings.html Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_settings.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update application/single_app/functions_documents.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Paul Lizer <paullizer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc * Control center (microsoft#567) * added group status (active, locked, upload disabled, and inactive) Adds Azure Billing Plugin in Community Customizations * added bulk member upload via csv for groups * add document metadata modified activity log tracking * activity logging for members deleted from groups * added group activity timeline * added notification system * added notifications for document upload to workspaces * fixed badge sizing * fixed url link * fixed badge to not show with zero notifications * Updated notification system * Updated approval system * updated approval workflow * updated notification workflow * Fixed set active bug on my public workspace page * Added user retention policy, updated user profile page with dashboards, retention config, and more. * adding speed to text for chat UI * updated the speech wave form and input field * updated to transcribe entire recording * fixed bug creating new conversation with auto-send * add mic permissions * added stream token tracking * Added public workspace reporting * Updated AI search sizing analysis * added management for public workspaces * improved public workspace management includes stats and bulk actions * updated groups dashboard for owners and admins with stats and bulk actions * added voice for ai to talk with users in chats * Auto Voice Response * for speech service, added 429 randomized response pattern to prevent thunder herding * updated admin settings for speech services and fixed dark mode for raw log viewing * updated video extraction card * Added Control Center Admin and Dashboard Reader roles * updated feedback and safety decorators so admins work unless required then those roles must be used * Updated and Validated logic for admin roles; control center, safety, and feedback * added support for control center admin and dashboard reader * Development (microsoft#566) * Banner text color picker from Vivien (microsoft#555) * Classification text color picker * Line endings * Remove opencensus * Add flask instrumentation * Add troubleshooting doc * Add troubleshooting doc --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * updated tool tip to better inform user on status of ai response * improve query parameters detection for swagger * updated visual cue showing the ai is talking to the user * moved duplicates to shared js * replaced alert with toast. * fixed and added log_event to exceptions * added @user_required and improved swagger generation * Update route_frontend_profile.py * fixed swagger generation bug on affecting two apis * returned keyvault to admin settings ui * Fixed bug when running local js --------- Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> * Adding release notes * fixed debug_debug_print * Updated README * Update README.md * accepted changes * removed files * GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app * Upated to v0.235.002 * removed debug test file * Updated to v0.235.003 * Update python-syntax-check.yml * fixed disable group creation bug * fixed bug not showing raw activity log for groups * fixed control center access to not require admin role when enabling controlcenteradmin role * fix documentation * Update release_notes.md * Update README.md * added click restrictions to top items in control center --------- Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com> Co-authored-by: Bionic711 <nadoyle@microsoft.com> Co-authored-by: cjackson202 <134412115+cjackson202@users.noreply.github.com> Co-authored-by: Bionic711 <ndoyle001@gmail.com> Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com> Co-authored-by: Steve Carroll <37545884+SteveCInVA@users.noreply.github.com> Co-authored-by: Xeelee33 <Xeelee33@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com> Co-authored-by: Ed Clark <clarked@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge in recent changes from Microsoft's Development branch