diff --git a/docs/ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md b/docs/ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md new file mode 100644 index 000000000..391c60ef4 --- /dev/null +++ b/docs/ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md @@ -0,0 +1,181 @@ +# GitHub Issue Response Template - DLP Impact Analysis Export Timeout + +This template can be used when responding to issues related to export timeouts when exporting DLP Impact Analysis data. + +--- + +## Template: DLP Impact Analysis Export Timeout + +**Use when:** Users report that they cannot export DLP Impact Analysis results due to timeouts or large dataset sizes + +**Response:** + +Thank you for reporting this issue with exporting DLP Impact Analysis data! + +### Issue Summary + +This is a **known limitation** when analyzing DLP policies with broad scope (such as "Strict" policies that block many connectors). The large number of impacted apps and flows creates too many records to export through the standard Canvas app or model-driven app export functions. + +### Root Cause + +The export timeouts are caused by platform limitations: +- **Canvas Apps**: 2,000 record delegation limit +- **Dataverse Excel Export**: 50,000 row limit +- **Browser Timeouts**: 30-120 seconds (varies by browser) +- **API Throttling**: Service protection limits for large queries + +When your DLP policy analysis generates more records than these limits, exports will timeout or fail. + +### Solutions Available + +We've created comprehensive troubleshooting documentation with **5 different solutions** to handle large DLP Impact Analysis exports: + +šŸ”— **[DLP Impact Analysis Export Timeout Troubleshooting Guide](../docs/troubleshooting/dlp-impact-analysis-export-timeout.md)** + +#### Quick Reference - Choose Your Approach: + +| Your Scenario | Recommended Solution | Complexity | +|--------------|---------------------|------------| +| < 2,000 records | Filter in Canvas App and export | āœ… Low | +| 2,000 - 50,000 records | Use filtered views in model-driven app | āœ… Low | +| > 50,000 records | **Power Automate batch export** | āš ļø Medium | +| Technical users | PowerShell with FetchXML pagination | šŸ”§ High | +| Analysis only (no export needed) | Power BI integration | āœ… Low-Medium | + +### Recommended Solution: Power Automate Flow Export + +For most users with large datasets, we recommend **Solution 1** from the troubleshooting guide: + +**Create a Power Automate flow** that: +1. Retrieves data in batches of 5,000 records +2. Exports to SharePoint/OneDrive or sends via email +3. Handles any dataset size without timeouts +4. Runs automatically in the background + +šŸ‘‰ [Step-by-step instructions for Power Automate export](../docs/troubleshooting/dlp-impact-analysis-export-timeout.md#solution-1-power-automate-flow-export-recommended-for-large-datasets) + +### For Technical Users: PowerShell Script + +We've also created a **ready-to-use PowerShell script** that automates the export: + +šŸ“„ **[Export-DLPImpactAnalysis.ps1](../docs/scripts/Export-DLPImpactAnalysis.ps1)** + +```powershell +# Install the required module +Install-Module Microsoft.Xrm.Data.PowerShell -Scope CurrentUser + +# Run the export +.\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://yourorg.crm.dynamics.com" +``` + +The script handles pagination automatically and exports to CSV. + +### Quick Workaround: Export Filtered Subsets + +If you need data immediately: + +1. Open the **Impact Analysis** app +2. Apply filters to reduce dataset size: + - Filter by **Environment** + - Filter by **Decision Status** (e.g., "Needs Review") + - Filter by **Date Range** (last 30 days) +3. Export the filtered subset (should be < 2,000 records) +4. Repeat for other filters to get complete dataset in parts + +### Next Steps + +1. Review the [complete troubleshooting guide](../docs/troubleshooting/dlp-impact-analysis-export-timeout.md) +2. Choose the solution that fits your scenario and technical skills +3. If you continue experiencing issues after trying these solutions, please provide: + - Approximate number of records in your DLP Impact Analysis + - Which solution you tried + - Any error messages or screenshots + - Your CoE Starter Kit version + +### Related Documentation + +- [CoE Starter Kit Documentation](https://learn.microsoft.com/power-platform/guidance/coe/starter-kit) +- [Dataverse Service Protection Limits](https://learn.microsoft.com/power-platform/admin/api-request-limits-allocations) +- [Power Apps Delegation Limits](https://learn.microsoft.com/power-apps/maker/canvas-apps/delegation-overview) + +### Prevention: Managing Large Datasets + +To prevent this issue in the future: + +āœ… **Run targeted analyses**: Analyze by environment rather than tenant-wide +āœ… **Clean up old data**: Delete completed analyses after 90 days +āœ… **Use Power BI**: Analyze directly without exporting + +Let us know if you have any questions or need additional guidance! + +--- + +## Template: Follow-up Questions + +**Use when:** The user provides more details or asks specific questions + +**Response:** + +Thank you for the additional information! + +[Address specific questions based on their scenario] + +### For Your Specific Situation + +**Dataset Size**: ~[X] records +**Recommended Approach**: [Choose from the solutions above] + +**Why this approach:** +- [Explain why it fits their needs] +- [Mention any considerations] + +**Estimated Time**: +- Setup: [X] minutes +- Execution: [X] minutes +- Total: [X] minutes + +### Additional Tips for Your Scenario + +[Provide scenario-specific guidance] + +Let me know if you need help with any of these steps! + +--- + +## Template: Issue Resolution + +**Use when:** Confirming the issue is resolved or providing final guidance + +**Response:** + +Great to hear [the solution worked / you're making progress]! + +### Summary + +āœ… **Issue**: Export timeout with large DLP Impact Analysis dataset +āœ… **Solution**: [The approach they used] +āœ… **Outcome**: Successfully exported [X] records + +### Documentation Updated + +This is a common issue, so we've added comprehensive troubleshooting documentation: +- [DLP Impact Analysis Export Timeout Guide](../docs/troubleshooting/dlp-impact-analysis-export-timeout.md) + +This should help other users encountering the same issue. + +### Feedback Welcome + +If you have suggestions for improving the documentation or additional solutions that worked for you, please share them! We're always looking to improve the CoE Starter Kit experience. + +Thank you for your patience and for helping improve the CoE Starter Kit! šŸŽ‰ + +--- + +## Notes for Maintainers + +- This issue typically occurs with "Strict" DLP policies that affect 10,000+ apps/flows +- The most common successful solution is Power Automate flow export (Solution 1) +- Technical users prefer the PowerShell script +- Some users may need guidance on how to estimate their record count first +- Consider adding telemetry to track how many records are in typical analyses +- Future enhancement: Add built-in batch export functionality to the Impact Analysis app diff --git a/docs/issue-response-templates.md b/docs/issue-response-templates.md index a2a3b3ec1..cc2965774 100644 --- a/docs/issue-response-templates.md +++ b/docs/issue-response-templates.md @@ -300,6 +300,41 @@ Let us know if the notifications continue after trying these steps! --- +## Template: DLP Impact Analysis Export Timeout + +**Use when:** Users report that they cannot export DLP Impact Analysis results due to timeouts or large dataset sizes + +**For the complete response template, see**: [ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md](ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md) + +**Quick Response:** + +Thank you for reporting this issue with exporting DLP Impact Analysis data! + +This is a **known limitation** when analyzing DLP policies with broad scope (such as "Strict" policies that block many connectors). The large number of impacted apps and flows creates too many records to export through standard methods. + +### Quick Reference + +We've created comprehensive documentation with **5 different solutions** based on your scenario: + +šŸ”— **[DLP Impact Analysis Export Timeout Troubleshooting Guide](troubleshooting/dlp-impact-analysis-export-timeout.md)** + +| Your Scenario | Recommended Solution | Complexity | +|--------------|---------------------|------------| +| < 2,000 records | Filter in Canvas App and export | āœ… Low | +| 2,000 - 50,000 records | Use filtered views in model-driven app | āœ… Low | +| > 50,000 records | **Power Automate batch export** | āš ļø Medium | +| Technical users | PowerShell with FetchXML pagination | šŸ”§ High | +| Analysis only | Power BI integration | āœ… Low-Medium | + +### Tools Available + +- **PowerShell Script**: [Export-DLPImpactAnalysis.ps1](scripts/Export-DLPImpactAnalysis.ps1) +- **Step-by-step Power Automate instructions**: [In troubleshooting guide](troubleshooting/dlp-impact-analysis-export-timeout.md#solution-1-power-automate-flow-export-recommended-for-large-datasets) + +Let me know if you need guidance on which approach would work best for your situation! + +--- + **Template Version**: 1.0 **Last Updated**: January 2026 **Maintained by**: CoE Starter Kit Community diff --git a/docs/scripts/Export-DLPImpactAnalysis.ps1 b/docs/scripts/Export-DLPImpactAnalysis.ps1 new file mode 100644 index 000000000..1aa448922 --- /dev/null +++ b/docs/scripts/Export-DLPImpactAnalysis.ps1 @@ -0,0 +1,240 @@ +<# +.SYNOPSIS + Exports DLP Impact Analysis data from Dataverse to CSV file. + +.DESCRIPTION + This script retrieves all records from the admin_dlpimpactanalysis table + in your CoE Starter Kit environment and exports them to a CSV file. + + The script handles pagination automatically and can export large datasets + that would timeout in the Canvas or model-driven apps. + +.PARAMETER EnvironmentUrl + The URL of your Dataverse environment (e.g., https://contoso.crm.dynamics.com) + +.PARAMETER OutputPath + The path where the CSV file will be saved. Defaults to current directory. + +.PARAMETER DLPPolicyName + Optional. Filter results by DLP Policy Name. If not specified, exports all records. + +.PARAMETER DaysBack + Optional. Only export records created in the last X days. If not specified, exports all records. + +.PARAMETER PageSize + Optional. Number of records to retrieve per page. Default is 5000. + +.EXAMPLE + .\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -OutputPath "C:\Exports\DLPImpact.csv" + + Exports all DLP Impact Analysis records to the specified file. + +.EXAMPLE + .\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -DLPPolicyName "Strict Policy" -OutputPath "StrictPolicy.csv" + + Exports only records for the "Strict Policy" DLP policy. + +.EXAMPLE + .\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -DaysBack 30 + + Exports only records created in the last 30 days. + +.NOTES + Prerequisites: + - Install Microsoft.Xrm.Data.PowerShell module: Install-Module Microsoft.Xrm.Data.PowerShell + - You must have read access to the admin_dlpimpactanalysis table in Dataverse + - You must have System Administrator or similar role in the CoE environment + + Version: 1.0 + Author: CoE Starter Kit Team + Date: 2024 +#> + +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$EnvironmentUrl, + + [Parameter(Mandatory = $false)] + [string]$OutputPath = "DLPImpactAnalysis_Export_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv", + + [Parameter(Mandatory = $false)] + [string]$DLPPolicyName, + + [Parameter(Mandatory = $false)] + [int]$DaysBack, + + [Parameter(Mandatory = $false)] + [int]$PageSize = 5000 +) + +# Check if Microsoft.Xrm.Data.PowerShell module is installed +$module = Get-Module -ListAvailable -Name Microsoft.Xrm.Data.PowerShell +if (-not $module) { + Write-Error "Microsoft.Xrm.Data.PowerShell module is not installed. Please install it using: Install-Module Microsoft.Xrm.Data.PowerShell" + exit 1 +} + +Import-Module Microsoft.Xrm.Data.PowerShell + +Write-Host "Connecting to Dataverse environment: $EnvironmentUrl" -ForegroundColor Cyan + +try { + # Connect to Dataverse + $conn = Get-CrmConnection -InteractiveMode + + if (-not $conn.IsReady) { + Write-Error "Failed to connect to Dataverse. Please check your credentials and environment URL." + exit 1 + } + + Write-Host "āœ“ Connected successfully" -ForegroundColor Green + + # Build FetchXML query + $filterConditions = "" + + if ($DLPPolicyName) { + $filterConditions += " `n" + } + + if ($DaysBack) { + $dateThreshold = (Get-Date).AddDays(-$DaysBack).ToString("yyyy-MM-dd") + $filterConditions += " `n" + } + + $fetchXml = @" + + + + + + + + + + + + + + + + + + + +$filterConditions + + + + +"@ + + Write-Host "`nFetch Query:" -ForegroundColor Cyan + Write-Host $fetchXml -ForegroundColor Gray + + Write-Host "`nRetrieving records..." -ForegroundColor Cyan + + $allResults = @() + $pageNumber = 1 + $moreRecords = $true + $totalRecords = 0 + + while ($moreRecords) { + Write-Host " Fetching page $pageNumber (up to $PageSize records)..." -ForegroundColor Yellow + + # Add page number to fetch XML + $pagedFetch = $fetchXml -replace 'count="' + $PageSize + '"', "count='$PageSize' page='$pageNumber'" + + try { + $response = Get-CrmRecordsByFetch -conn $conn -Fetch $pagedFetch + + if ($response.CrmRecords.Count -gt 0) { + $allResults += $response.CrmRecords + $totalRecords += $response.CrmRecords.Count + Write-Host " Retrieved $($response.CrmRecords.Count) records (Total: $totalRecords)" -ForegroundColor Green + } + + # Check if there are more records + if ($response.CrmRecords.Count -lt $PageSize) { + $moreRecords = $false + Write-Host " āœ“ All records retrieved" -ForegroundColor Green + } + else { + $pageNumber++ + } + } + catch { + Write-Error "Error retrieving records: $_" + exit 1 + } + } + + if ($totalRecords -eq 0) { + Write-Host "`nāš  No records found matching the specified criteria." -ForegroundColor Yellow + exit 0 + } + + Write-Host "`nāœ“ Retrieved $totalRecords total records" -ForegroundColor Green + Write-Host "`nExporting to CSV: $OutputPath" -ForegroundColor Cyan + + # Convert to custom objects for better CSV export + $exportData = $allResults | ForEach-Object { + [PSCustomObject]@{ + 'ID' = $_.admin_dlpimpactanalysisid + 'Name' = $_.admin_name + 'DLP Policy Name' = $_.admin_dlppolicyname + 'Decision' = $_.admin_decision + 'Decision Status' = $_.admin_decisionstatus + 'Conflicting Connector (Blocked)' = $_.admin_conflictingconnectorblocked + 'Conflicting Connector (Business)' = $_.admin_conflictingconnectorbusiness + 'Conflicting Connector (Non-Business)' = $_.admin_conflictingconnectornonbusiness + 'Impacted App' = $_.admin_impactedapp + 'Impacted Flow' = $_.admin_impactedflow + 'Business Impact' = $_.admin_businessimpact + 'Action Required On' = $_.admin_actionrequiredon + 'Created On' = $_.createdon + 'Modified On' = $_.modifiedon + 'State' = $_.statecode + } + } + + # Export to CSV + $exportData | Export-Csv -Path $OutputPath -NoTypeInformation -Encoding UTF8 + + Write-Host "āœ“ Export completed successfully!" -ForegroundColor Green + Write-Host "`nExport Summary:" -ForegroundColor Cyan + Write-Host " Total Records: $totalRecords" + Write-Host " Output File: $OutputPath" + Write-Host " File Size: $([math]::Round((Get-Item $OutputPath).Length / 1MB, 2)) MB" + + # Provide summary statistics + Write-Host "`nRecord Statistics:" -ForegroundColor Cyan + $uniquePolicies = ($exportData | Select-Object -ExpandProperty 'DLP Policy Name' -Unique).Count + Write-Host " Unique DLP Policies: $uniquePolicies" + + $impactedApps = ($exportData | Where-Object { $_.'Impacted App' } | Measure-Object).Count + $impactedFlows = ($exportData | Where-Object { $_.'Impacted Flow' } | Measure-Object).Count + Write-Host " Impacted Apps: $impactedApps" + Write-Host " Impacted Flows: $impactedFlows" + + if ($DLPPolicyName) { + Write-Host " Filtered by Policy: $DLPPolicyName" + } + if ($DaysBack) { + Write-Host " Filtered by Date: Last $DaysBack days" + } + + Write-Host "`nāœ“ Done!" -ForegroundColor Green +} +catch { + Write-Error "An error occurred: $_" + Write-Error $_.Exception.Message + Write-Error $_.ScriptStackTrace + exit 1 +} +finally { + # Clean up connection + if ($conn) { + $conn.Dispose() + } +} diff --git a/docs/scripts/README.md b/docs/scripts/README.md new file mode 100644 index 000000000..d40598aea --- /dev/null +++ b/docs/scripts/README.md @@ -0,0 +1,91 @@ +# CoE Starter Kit Scripts + +This directory contains PowerShell and other scripts to help with common CoE Starter Kit administration and maintenance tasks. + +## Available Scripts + +### Export-DLPImpactAnalysis.ps1 + +**Purpose**: Export DLP Impact Analysis data from Dataverse to CSV file when the Canvas app or model-driven app export times out. + +**Use Case**: When you have a large number of impacted apps/flows from DLP policy analysis (e.g., > 2,000 records), this script retrieves and exports all data using pagination. + +**Prerequisites**: +- PowerShell 5.1 or higher +- Microsoft.Xrm.Data.PowerShell module: `Install-Module Microsoft.Xrm.Data.PowerShell` +- System Administrator or equivalent role in the CoE environment + +**Usage Examples**: + +```powershell +# Export all DLP Impact Analysis records +.\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" + +# Export records for a specific DLP policy +.\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -DLPPolicyName "Strict Policy" + +# Export records from the last 30 days only +.\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -DaysBack 30 + +# Export to a specific file location +.\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://contoso.crm.dynamics.com" -OutputPath "C:\Exports\DLPImpact.csv" +``` + +**Related Documentation**: See [DLP Impact Analysis Export Timeout Troubleshooting](../troubleshooting/dlp-impact-analysis-export-timeout.md) for complete guidance on handling large exports. + +## Installation + +### Installing Microsoft.Xrm.Data.PowerShell + +Most scripts in this directory require the Microsoft.Xrm.Data.PowerShell module. Install it using: + +```powershell +Install-Module Microsoft.Xrm.Data.PowerShell -Scope CurrentUser +``` + +If you encounter issues, you may need to update PowerShellGet first: + +```powershell +Install-Module PowerShellGet -Force -Scope CurrentUser +``` + +### Execution Policy + +If you receive an execution policy error, you may need to adjust your PowerShell execution policy: + +```powershell +# View current policy +Get-ExecutionPolicy + +# Set to RemoteSigned (recommended for scripts) +Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser +``` + +## Security Considerations + +- **Credentials**: Scripts use interactive authentication. Never hardcode credentials in scripts. +- **Data Export**: Exported CSV files may contain sensitive information. Store them securely and delete when no longer needed. +- **Execution**: Only run scripts from trusted sources. Review script contents before execution. +- **Permissions**: Scripts require appropriate permissions in your environment. Use least-privilege access where possible. + +## Contributing + +If you've created a useful script for CoE Starter Kit administration: + +1. Ensure it follows PowerShell best practices +2. Add comprehensive comment-based help +3. Test thoroughly in a non-production environment +4. Add an entry to this README +5. Submit a pull request + +See [HOW_TO_CONTRIBUTE.md](../../HOW_TO_CONTRIBUTE.md) for more information. + +## Getting Help + +- **Script Issues**: Report issues on [GitHub Issues](https://github.com/microsoft/coe-starter-kit/issues) +- **General Questions**: Ask in the [Power Apps Community Forum](https://powerusers.microsoft.com/t5/Power-Apps-Governance-and/bd-p/Admin_PowerApps) +- **Documentation**: See [CoE Starter Kit Documentation](https://learn.microsoft.com/en-us/power-platform/guidance/coe/starter-kit) + +## Disclaimer + +These scripts are provided as-is without warranty. Always test scripts in a development or test environment before running in production. Review and understand what each script does before execution. diff --git a/docs/troubleshooting/README.md b/docs/troubleshooting/README.md index ad50b95e4..84a9ff173 100644 --- a/docs/troubleshooting/README.md +++ b/docs/troubleshooting/README.md @@ -4,9 +4,6 @@ This directory contains troubleshooting guides for common issues encountered whe ## Available Guides -### Authentication Issues - -- **[DLP Impact Analysis Authentication Error](../ISSUE-RESPONSE-DLP-Impact-Analysis-Authentication.md)** - Resolve "UserNotLoggedIn" and "untrusted authority" errors when opening custom pages in model-driven apps, specifically the Data Policy Impact Analysis app. ### Power BI Connection Issues diff --git a/docs/troubleshooting/dlp-impact-analysis-export-timeout-quickref.md b/docs/troubleshooting/dlp-impact-analysis-export-timeout-quickref.md new file mode 100644 index 000000000..5e88fb209 --- /dev/null +++ b/docs/troubleshooting/dlp-impact-analysis-export-timeout-quickref.md @@ -0,0 +1,74 @@ +# DLP Impact Analysis Export Timeout - Quick Reference + +**Problem**: Cannot export DLP Impact Analysis results due to timeouts when analyzing strict DLP policies + +**Reason**: Large number of impacted records exceeds platform limitations (Canvas app delegation, browser timeouts, export limits) + +## Quick Solutions + +### 1. šŸ“Š Power Automate Flow (Best for Large Datasets) +- **Best for**: > 2,000 records +- **Time**: 30-60 min to setup +- **Complexity**: Medium +- **Steps**: Create flow ā†’ List rows in batches ā†’ Export to SharePoint/Email +- **Details**: [Full instructions](dlp-impact-analysis-export-timeout.md#solution-1-power-automate-flow-export-recommended-for-large-datasets) + +### 2. šŸ’» PowerShell Script (For Technical Users) +- **Best for**: Any size dataset +- **Time**: 5 min to setup + run time +- **Complexity**: High (requires PowerShell knowledge) +- **Script**: [Export-DLPImpactAnalysis.ps1](../scripts/Export-DLPImpactAnalysis.ps1) +- **Usage**: + ```powershell + Install-Module Microsoft.Xrm.Data.PowerShell + .\Export-DLPImpactAnalysis.ps1 -EnvironmentUrl "https://yourorg.crm.dynamics.com" + ``` + +### 3. šŸ” Filter and Export Subsets (Quick Workaround) +- **Best for**: < 2,000 records per export +- **Time**: 5 min +- **Complexity**: Low +- **Steps**: + 1. Open Impact Analysis app + 2. Apply filters (Environment, Date, Status) + 3. Export filtered subset + 4. Repeat with different filters + +### 4. šŸ“ˆ Power BI (No Export Needed) +- **Best for**: Analysis without export +- **Time**: 15-30 min +- **Complexity**: Low-Medium +- **Steps**: Connect Power BI Desktop ā†’ Select admin_dlpimpactanalysis table ā†’ Create reports + +### 5. šŸ“ FetchXML with Pagination (For Developers) +- **Best for**: Custom integrations +- **Time**: 60+ min +- **Complexity**: High +- **Details**: [FetchXML instructions](dlp-impact-analysis-export-timeout.md#solution-2-fetchxml-with-pagination-for-developers) + +## Decision Matrix + +| Record Count | Recommended Solution | Alternative | +|--------------|---------------------|-------------| +| < 2,000 | Filter in Canvas App (Solution 3) | Power BI (Solution 4) | +| 2,000 - 10,000 | Power Automate (Solution 1) | Filtered exports (Solution 3) | +| 10,000 - 50,000 | Power Automate (Solution 1) | PowerShell (Solution 2) | +| > 50,000 | PowerShell (Solution 2) | Power Automate (Solution 1) | + +## Need More Help? + +- **Full Documentation**: [dlp-impact-analysis-export-timeout.md](dlp-impact-analysis-export-timeout.md) +- **PowerShell Script**: [Export-DLPImpactAnalysis.ps1](../scripts/Export-DLPImpactAnalysis.ps1) +- **Issue Response Template**: [ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md](../ISSUE-RESPONSE-dlp-impact-analysis-export-timeout.md) +- **GitHub Issues**: [Report a problem](https://github.com/microsoft/coe-starter-kit/issues) + +## Prevention Tips + +āœ… Run targeted analyses (by environment, not tenant-wide) +āœ… Clean up old analysis records regularly (90-day retention) +āœ… Use Power BI for ongoing monitoring instead of repeated exports +āœ… Test strict policies on subset of environments first + +--- +**Last Updated**: February 2024 +**Related**: CoE Starter Kit Core Components v4.50.8+ diff --git a/docs/troubleshooting/dlp-impact-analysis-export-timeout.md b/docs/troubleshooting/dlp-impact-analysis-export-timeout.md new file mode 100644 index 000000000..fa4b33b38 --- /dev/null +++ b/docs/troubleshooting/dlp-impact-analysis-export-timeout.md @@ -0,0 +1,363 @@ +# DLP Impact Analysis Export Timeout Troubleshooting + +## Issue Description + +When running the Impact Analysis app for DLP policies with large scope (e.g., "Strict" policies affecting many apps and flows), the export operation may timeout or fail to complete. This occurs when attempting to export results from: + +- The Impact Analysis Canvas app +- The model-driven app using the `admin_dlpimpactanalysis` Dataverse table + +The issue is caused by the large volume of records returned, which exceeds: +- Canvas app delegation limits (2,000 records by default) +- Browser timeout limits for large exports +- Dataverse export size limitations (50,000 rows for Excel exports) +- API throttling limits when retrieving large datasets + +## Affected Components + +- **Solution**: Core Components (v4.50.8 and earlier) +- **App**: Impact Analysis (Canvas App) +- **Entity**: `admin_dlpimpactanalysis` (DLP Impact Analysis table) +- **Model-driven App**: DLP Impact Analysis + +## Root Cause + +The `admin_dlpimpactanalysis` table stores one record per impacted app or flow for each DLP policy analysis. When analyzing a "Strict" DLP policy that blocks most connectors: + +1. A large number of apps and flows in your tenant may be impacted +2. Each impacted resource creates a separate record in the table +3. Export operations attempt to retrieve all records at once +4. Large datasets exceed timeout limits and delegation constraints + +### Technical Limitations + +| Component | Limitation | Impact | +|-----------|------------|--------| +| Canvas Apps | 2,000 record delegation limit | Cannot display or export more than 2,000 records without pagination | +| Dataverse Export (Excel) | 50,000 row limit | Large exports may fail or be truncated | +| Browser Timeout | 30-120 seconds (varies) | Long-running exports timeout before completion | +| API Throttling | Service protection limits | Excessive API calls may be throttled | + +## Workarounds and Solutions + +### Solution 1: Power Automate Flow Export (Recommended for Large Datasets) + +Create a Power Automate cloud flow to export the data in batches and send it via email or store it in SharePoint/OneDrive. + +#### Step-by-Step Instructions + +1. **Create a new Manual trigger flow** in your CoE environment: + - Go to Power Automate (https://make.powerautomate.com) + - Select your CoE environment + - Create a new **Instant cloud flow** + - Add a **Manual trigger** + +2. **Add variables for configuration**: + - Add **Initialize variable** actions: + - `varDLPPolicyName` (String) - The name of the DLP policy to export + - `varBatchSize` (Integer) - Set to 5000 for batch processing + - `varSkipCount` (Integer) - Initialize to 0 + +3. **Add a Do Until loop** to process data in batches: + ``` + Do Until: varSkipCount is greater than 100000 + ``` + +4. **Inside the loop, add List rows action**: + - **Table name**: DLP Impact Analyses (admin_dlpimpactanalysis) + - **Row count**: Use `varBatchSize` (5000) + - **Skip token**: Use `varSkipCount` + - **Filter rows** (optional): `admin_dlppolicyname eq 'YOUR_POLICY_NAME'` + - **Select columns** (optional): Choose specific columns to reduce data size + +5. **Create CSV/Excel from batch**: + - Use **Create CSV table** or **Create HTML table** action + - Input: `body('List_rows')?['value']` + +6. **Append to file** (if using SharePoint/OneDrive): + - **Action**: Append to file (SharePoint/OneDrive) + - **File name**: `DLPImpactAnalysis_Export.csv` + - **Content**: Output from CSV table action + +7. **Increment skip count**: + - **Increment variable**: `varSkipCount` + - **Value**: `add(variables('varSkipCount'), variables('varBatchSize'))` + +8. **Check if more records exist**: + - Add **Condition**: `length(body('List_rows')?['value'])` is less than `varBatchSize` + - If true, **Terminate** with status Success + +9. **Send completion email** (after loop completes): + - **Action**: Send an email (V2) + - **To**: Your email + - **Subject**: DLP Impact Analysis Export Complete + - **Body**: Include file link or attach the export + +#### Sample Flow Template + +```json +{ + "description": "Export DLP Impact Analysis data in batches", + "trigger": "Manual", + "actions": [ + "Initialize varDLPPolicyName", + "Initialize varBatchSize (5000)", + "Initialize varSkipCount (0)", + "Do Until (varSkipCount > 100000)", + " - List rows (admin_dlpimpactanalysis)", + " - Create CSV table", + " - Append to file (SharePoint)", + " - Increment varSkipCount", + " - Check if complete", + "Send completion email with file" + ] +} +``` + +### Solution 2: FetchXML with Pagination (For Developers) + +Use FetchXML queries with pagination to retrieve data programmatically. + +#### Using Power Automate + +1. **Create a flow** with **Manual trigger** + +2. **Add Dataverse action**: **Perform an unbound action** + - **Action Name**: `RetrieveMultiple` + +3. **Use FetchXML with pagination**: + +```xml + + + + + + + + + + + + + + + + + + +``` + +4. **Process results in batches** and combine into a single file + +#### Using PowerShell / C# + +For technical users, use the Dataverse SDK or Web API to retrieve data with pagination: + +**PowerShell Example:** +```powershell +# Install-Module Microsoft.Xrm.Data.PowerShell + +$conn = Get-CrmConnection -Interactive + +$fetchXml = @" + + + + + + + + + +"@ + +$results = @() +$pageNumber = 1 +$moreRecords = $true + +while ($moreRecords) { + $pagedFetch = $fetchXml -replace 'count="5000"', "count='5000' page='$pageNumber'" + $response = Get-CrmRecordsByFetch -conn $conn -Fetch $pagedFetch + + $results += $response.CrmRecords + + if ($response.CrmRecords.Count -lt 5000) { + $moreRecords = $false + } else { + $pageNumber++ + } +} + +# Export to CSV +$results | Export-Csv -Path "DLPImpactAnalysis.csv" -NoTypeInformation +``` + +### Solution 3: Filter and Export Subsets + +Instead of exporting all records at once, filter by specific criteria to reduce the dataset size. + +#### In the Impact Analysis App + +1. Open the **Impact Analysis** app +2. Apply filters before export: + - **DLP Policy**: Select the specific policy + - **Decision Status**: Filter by "Needs Review" or other status + - **Environment**: Filter by specific environment + - **Date Range**: Filter by creation date + +3. Export the filtered subset (should be under 2,000 records) + +#### In the Model-Driven App + +1. Open the **DLP Impact Analysis** model-driven app +2. Create a **Personal View** with filters: + - Navigate to **Advanced Find** + - Add filter conditions: + - DLP Policy Name equals "[Your Policy]" + - Decision Status equals specific value + - Created On in last X days + - Save as Personal View + +3. Apply the view and export (should be under 50,000 rows) + +### Solution 4: Power BI for Analysis (No Export Needed) + +Instead of exporting data, use Power BI to analyze the DLP impact directly from Dataverse. + +#### Steps: + +1. Open **Power BI Desktop** + +2. **Get Data** from **Dataverse**: + - Connect to your CoE environment URL + - Select the `admin_dlpimpactanalysis` table + +3. **Apply filters** in Power Query: + - Filter by `admin_dlppolicyname` + - Filter by `statecode` = 0 (Active) + +4. **Create visualizations**: + - Table of impacted apps and flows + - Count by decision status + - Count by environment + - Count by maker + +5. **Publish to Power BI Service** for sharing with stakeholders + +This approach: +- Handles unlimited records +- Provides interactive analysis +- Allows drill-down without exporting +- Can be refreshed on a schedule + +### Solution 5: Optimize the Analysis Scope + +Prevent the large dataset by being more targeted in your DLP impact analysis. + +#### Recommendations: + +1. **Analyze by Environment**: Instead of analyzing the policy globally, run separate analyses for each environment + +2. **Use Test Environments**: Test strict policies on a subset of environments first + +3. **Clean Up Old Analyses**: Delete old DLP Impact Analysis records before running new analyses: + - Go to **Settings** ā†’ **Advanced Settings** ā†’ **Data Management** + - Create a **Bulk Delete** job for old `admin_dlpimpactanalysis` records + +4. **Limit Policy Scope**: Consider applying strict DLP policies to specific environments rather than tenant-wide + +## Best Practices + +### Before Running DLP Impact Analysis + +1. **Estimate Impact**: Use the Dataverse table directly to count records: + ``` + Filter: admin_dlppolicyname eq 'PolicyName' and statecode eq 0 + Count: admin_dlpimpactanalysisid + ``` + +2. **Clean Up Previous Analyses**: Delete old analysis records to improve performance + +3. **Plan for Large Exports**: If you expect > 5,000 impacted resources, prepare to use Power Automate or PowerShell + +### After Running DLP Impact Analysis + +1. **Export Immediately**: Don't wait if the dataset is manageable (< 2,000 records) + +2. **Use Filtered Views**: Create multiple exports with different filters rather than one large export + +3. **Archive Data**: Move completed analysis data to external storage (SharePoint, Azure Blob Storage) for long-term retention + +## Prevention and Optimization + +### Table Management + +To prevent the `admin_dlpimpactanalysis` table from growing too large: + +1. **Enable Data Retention Policies**: Configure Dataverse data retention for this table + - Retain records for 90 days + - Auto-delete completed analyses older than retention period + +2. **Manual Cleanup Flow**: Create a scheduled flow to delete old records: + - Run weekly + - Delete records older than 90 days + - Filter by `statecode eq 0` and `createdon le [90 days ago]` + +### Analysis Strategy + +1. **Incremental Analysis**: Instead of one large analysis, run multiple smaller analyses: + - By environment + - By business unit + - By maker + +2. **Periodic Review**: Schedule regular DLP impact reviews rather than ad-hoc large analyses + +## Related Documentation + +- [CoE Starter Kit Documentation](https://learn.microsoft.com/power-platform/guidance/coe/starter-kit) +- [Dataverse Service Protection Limits](https://learn.microsoft.com/power-platform/admin/api-request-limits-allocations) +- [Power Apps Delegation Limits](https://learn.microsoft.com/power-apps/maker/canvas-apps/delegation-overview) +- [FetchXML Pagination](https://learn.microsoft.com/power-apps/developer/data-platform/org-service/page-large-result-sets-with-fetchxml) + +## Additional Resources + +### Sample Power Automate Flow + +Sample flows for exporting DLP Impact Analysis data are documented in detail in the troubleshooting guide above. Follow the step-by-step instructions in **Solution 1** to create your own flow. + +### PowerShell Script + +A complete PowerShell script for bulk export is available in the repository: +- Location: [docs/scripts/Export-DLPImpactAnalysis.ps1](../scripts/Export-DLPImpactAnalysis.ps1) +- Usage instructions: [docs/scripts/README.md](../scripts/README.md) + +## Need More Help? + +If you continue to experience issues after trying these solutions: + +1. **Check your Dataverse storage capacity**: Large tables may be throttled if storage is near capacity + +2. **Verify service health**: Check the [Microsoft 365 Service Health Dashboard](https://admin.microsoft.com/Adminportal/Home#/servicehealth) + +3. **Contact support**: For persistent issues, contact Microsoft Support with: + - Error messages and screenshots + - Record count in `admin_dlpimpactanalysis` table + - Export method being used + - CoE Starter Kit version + +4. **Create a GitHub issue**: Report the issue at https://github.com/microsoft/coe-starter-kit/issues with full details + +## Summary + +Export timeouts for DLP Impact Analysis data are caused by large datasets exceeding platform limitations. Use one of these approaches based on your scenario: + +| Scenario | Recommended Solution | Complexity | +|----------|---------------------|------------| +| < 2,000 records | Filter in Canvas App and export | Low | +| 2,000 - 50,000 records | Use filtered views in model-driven app | Low | +| > 50,000 records | Power Automate batch export | Medium | +| Technical users | PowerShell with FetchXML pagination | High | +| Analysis only (no export) | Power BI integration | Low-Medium | + +For most users dealing with large datasets, **Solution 1 (Power Automate Flow Export)** is the recommended approach as it handles any dataset size and requires no technical expertise.