Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 27, 2026

Admin sync flows (v3 Flow Action Details, v4 Flows) running 7+ hours instead of <1 hour due to API throttling from concurrent executions. Root cause: Driver updates trigger 20+ child flows simultaneously; without delay settings, creates 1000+ concurrent API calls exceeding Dataverse limits (6K req/5min), forcing retry cascades with exponential backoff.

Changes

Core Documentation

  • TROUBLESHOOTING-SYNC-PERFORMANCE.md - Comprehensive guide covering symptoms, root causes, 6 solution strategies, performance benchmarks by tenant size, and advanced troubleshooting
  • ISSUE-ANALYSIS-SYNC-PERFORMANCE.md - Technical analysis of cascading trigger pattern, concurrency settings (repetitions: 50 in loops), and throttling behavior

User Guidance

  • ISSUE-RESPONSE-SYNC-PERFORMANCE.md - Response template for similar issues with expected performance tables and follow-up steps
  • Updated Core Components README, troubleshooting directory README, and main README with links to new guides

Solution

Enable environment variables:

  • admin_DelayObjectInventory = Yes (adds 0-2s random delays in object inventory)
  • admin_DelayInventory = Yes (spaces out environment processing in Driver)

Expected Performance (delays enabled, incremental mode):

  • Small (<500 flows): 10-20 min
  • Medium (500-2K flows): 30-60 min
  • Large (2K-10K flows): 1-2 hrs
  • Enterprise (10K+ flows): 2-4 hrs

Technical Details

Flow architecture creates multiplicative concurrency:

Driver → updates admin_environment table
  ↓
Triggers 20+ flows via Dataverse webhook (message: 4, scope: 4)
  ↓  
Each flow: foreach with concurrency.repetitions: 50
  ↓
Result: 20 × 50 = 1000+ concurrent API calls → throttling → retries → 7+ hour runtime

No code changes required. Configuration issue, not bug.

Original prompt

This section details on the original issue you should resolve

<issue_title>[CoE Starter Kit - BUG] ISSUE TITLE</issue_title>
<issue_description>### Does this bug already exist in our backlog?

  • I have checked and confirm this is a new bug.

Describe the issue

Admin | Sync Template v3 (Flow Action Details) in this for some environments it is in Running state for long more than 7 hrs.
Also the
Admin | Sync Template v4 (Flows) . Both are triggered on the environment table change and might be that is causing the throttle..

Expected Behavior

Should complete the flow in max 1 hour-...as before

What solution are you experiencing the issue with?

Core

What solution version are you using?

4.50.6

What app or flow are you having the issue with?

Admin | Sync Template v3 (Flow Action Details)

What method are you using to get inventory and telemetry?

None

Steps To Reproduce

No response

Anything else?

No response</issue_description>

<agent_instructions>Need root cause analysis on taking time</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Custom agent used: CoE Custom Agent
Specialized agent for triaging and fixing issues in the CoE Starter Kit. Always consult the official CoE docs and the team’s SharePoint notes first, then mine prior GitHub issues and the CoE Common Responses playbook.


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits January 27, 2026 05:49
Co-authored-by: mohamrizwa <220306360+mohamrizwa@users.noreply.github.com>
…uides

Co-authored-by: mohamrizwa <220306360+mohamrizwa@users.noreply.github.com>
Co-authored-by: mohamrizwa <220306360+mohamrizwa@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix long running state in Sync Template v3 and v4 Add documentation for sync template throttling and performance issues Jan 27, 2026
Copilot AI requested a review from mohamrizwa January 27, 2026 05:52
@mohamrizwa mohamrizwa marked this pull request as ready for review February 4, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CoE Starter Kit - BUG] ISSUE TITLE

2 participants