Add data refresh capability for original tables with automatic derived table regeneration

[Copilot]

Users need to update base data after creating charts without losing derived transformations. Currently no mechanism exists to refresh data and propagate changes to dependent tables.

Changes

Backend (tables_routes.py)

• Added /api/tables/refresh-derived-data endpoint that re-executes Python transformation code in subprocess for security
• Validates column consistency between original and updated data
• Returns refreshed data for all dependent derived tables

Frontend (DataThread.tsx)

• Collapsed delete/attach-metadata actions into MoreVert menu (keeps generate-chart button separate per requirements)
• Added "Refresh data" menu option for original tables only (derived tables excluded)
• Implemented RefreshDataDialog component supporting file upload (CSV/XLSX/JSON) or raw JSON paste
• Enhanced validation to check all rows for required columns, not just first row
• Fixed derived table filtering to handle both string and array source types

State Management (dfSlice.tsx)

• Added updateTableRows action to update table data and refresh concept shelf items

Usage

// User workflow
1. Click MoreVert menu on original table → "Refresh data"
2. Upload new file or paste JSON (columns must match existing schema)
3. System validates, replaces base table, re-executes all derived table transformations
4. Redux state updates, UI reflects new data across all tables

Security

Python transformations execute in subprocess with audit hooks blocking file writes and dangerous operations.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature request: add an option to refresh data after chart creation</issue_title>
<issue_description>Allow users to make minor updates to data (additional rows, or update some values) after some charts have been created.

Additionally, have an option to auto referesh data if data is streamed from an external source. </issue_description>

<agent_instructions>Add a button in the data thread allowing user to replace an existing data with updated data, requirement: the updated data needs to have the same field names as the current data, and we are only allowed to replace original data (not derived data). When the new data is replaced, automatically re-run generate python code on the server to re-generate new derived data based on the updated table.

requirements:

1. collapse the new bottom with existing delete/attach metadata buttons into a menu (don't collapse generate chart button) so it's compact in the UI
2. implement a function in the server for refresh derived data, by providing updated data with their corresponding execution python code, this ececution should be run in subprocess</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Feature request: add an option to refresh data after chart creation #224

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.