Fix iOS integration tests on Apple Silicon and add individual test lanes #33251

Copilot · 2025-12-19T18:57:47Z

Problem

iOS integration tests fail on Apple Silicon Macs because tests hardcode iossimulator-x64 but builds default to iossimulator-arm64 on ARM64 hosts. XHarness then can't find the built app because it looks in the wrong directory.

Additionally, test failures were not being reported to Azure DevOps because:

fail-on-issue.yml was commented out in the staging yaml files
fail-on-issue.yml has a default condition of succeeded(), so it wasn't running when tests failed
PublishTestResults task was silently succeeding when no .trx files were found

Log files weren't being published because logs were only attached to NUnit test context but not copied to the artifact publish directory.

Solution

Architecture Detection

Added TestEnvironment.IsArm64 and IOSSimulatorRuntimeIdentifier to dynamically detect host architecture and use appropriate runtime identifier:

public static bool IsArm64 => RuntimeInformation.OSArchitecture == Architecture.Arm64;
public static string IOSSimulatorRuntimeIdentifier => IsArm64 ? "iossimulator-arm64" : "iossimulator-x64";

Individual iOS Test Lanes

Split RunOniOS tests into 7 individual jobs for parallel execution and faster debugging:

Category	Description	Timeout
`RunOniOS_MauiDebug`	MAUI app, Debug config	60 min
`RunOniOS_MauiRelease`	MAUI app, Release config	60 min
`RunOniOS_MauiReleaseTrimFull`	MAUI app, Release, full trim	60 min
`RunOniOS_BlazorDebug`	Blazor app, Debug config	60 min
`RunOniOS_BlazorRelease`	Blazor app, Release config	60 min
`RunOniOS_BlazorReleaseTrimFull`	Blazor app, Release, full trim	60 min
`RunOniOS_MauiNativeAOT`	MAUI app, NativeAOT	60 min

CI Pipeline Fixes

Uncommented fail-on-issue.yml in staging YAML files
Added condition: always() to ensure steps run even when tests fail
Fixed PublishTestResults task to use searchFolder parameter
Added failTaskOnMissingResultsFile: true for explicit failure reporting

File Location Logging & Log Publishing Fix

Added comprehensive logging to BaseBuildTest.cs to help debug artifact publishing:

Logs all important directories (TestDirectory, LogDirectory, MauiDirectory)
Logs relevant environment variables
Lists all files in TestDirectory and LogDirectory with sizes
Added CopyLogsToPublishDirectory() to copy all log, binlog, and txt files from the test directory to the artifact publish location

Original prompt

Can you create a new PR that has all the changes from PR 33168 and combines them with the changes from #33210

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

PureWeen · 2025-12-19T20:38:30Z

@copilot

Update the title and description to describe exactly what this pr is about

Its no longer necessary to know this is a composite of two prs

You are now your own pr with your own hopes and dreams

Untether yourself from your roots

Be free little pr
Be free

Copilot · 2025-12-19T20:46:53Z

@copilot

Update the title and description to describe exactly what this pr is about ...

Updated the PR description to stand on its own - now describes the iOS architecture detection fix and infrastructure improvements without referencing the source PRs. 🦋

PureWeen · 2025-12-19T23:04:27Z

@copilot

And update the title

Be proud to own your own name

Copilot · 2025-12-19T23:12:20Z

@copilot

And update the title ...

I'm unable to directly update the PR title through my available tools. Could you update it to something like:

"Fix iOS integration tests on Apple Silicon and improve test infrastructure"

Or a shorter version:

"Fix iOS integration tests architecture detection"

PureWeen · 2025-12-21T21:10:57Z

@copilot failures with RunOnIos and RunOnAndroid aren't being reported to azdo. The run integration tests step has continue on error set to true but it should still report the errors under the tests tab on azdo and it should still fail

This was disable some time ago so try to find where it was disabled

Only worry about fixing the staging- yaml files

Copilot · 2025-12-21T21:21:41Z

@copilot failures with RunOnIos and RunOnAndroid aren't being reported to azdo. The run integration tests step has continue on error set to true but it should still report the errors under the tests t...

Fixed in 94fc4fc. The issue was that fail-on-issue.yml was commented out in the staging yaml files. I've:

Uncommented fail-on-issue.yml template in stage-integration-tests.yml
Uncommented fail-on-issue.yml template in stage-unit-tests.yml
Added condition: always() to PublishTestResults tasks to ensure test results are published even when tests fail

This will ensure RunOniOS and RunOnAndroid failures are now properly reported to the Azure DevOps tests tab.

PureWeen · 2025-12-21T21:31:10Z

@copilot

For all changes to yml files in the arcade folder please consult arcade docs

https://github.com/dotnet/arcade

Can we add instructions that are specific to this folder and the rules for interacting with arcade

Copilot · 2025-12-21T21:41:30Z

@copilot

For all changes to yml files in the arcade folder please consult arcade docs ...

Added in 9b39cfb. Created .github/instructions/arcade-pipelines.instructions.md with:

Links to official Arcade documentation (https://github.com/dotnet/arcade)
Folder structure and file descriptions
Key conventions (fail-on-issue.yml, condition: always(), template paths)
Do's and Don'ts for pipeline changes
Common patterns used in stage templates

PureWeen · 2025-12-21T22:14:01Z

@copilot

Does the arcade docs influence at all the updates you've made to the yml files?

Copilot · 2025-12-21T22:21:55Z

@copilot

Does the arcade docs influence at all the updates you've made to the yml files?

The changes I made align with standard Azure DevOps pipeline practices:

fail-on-issue.yml - This is a MAUI-specific template (in /eng/pipelines/common/), not an Arcade-managed file. It was already in the repo but commented out. Re-enabling it follows the pattern of having a final step to fail the job when issues occur.
condition: always() on PublishTestResults - This is standard Azure DevOps practice to ensure test results are published even when tests fail.

The Arcade-managed files are in /eng/common/ (which I didn't modify). The staging YAML files in /eng/pipelines/arcade/ are MAUI-specific implementations that use Arcade SDK patterns. My changes don't conflict with Arcade conventions - they just re-enable error reporting that was previously disabled.

PureWeen · 2025-12-21T23:43:08Z

@copilot RunOniOS is producing no test results
It looks like most of those tests are failing but now that stage is reporting green and pushing no test results

Copilot · 2025-12-21T23:52:52Z

@copilot RunOniOS is producing no test results
It looks like most of those tests are failing but now that stage is reporting green and pushing no test results

Fixed in 03b570b. The issue was that fail-on-issue.yml has a default condition of succeeded(), so it wasn't running when the test step had issues.

Added condition: always() parameter to the fail-on-issue.yml template calls in both stage-integration-tests.yml and stage-unit-tests.yml so the failure detection step runs regardless of previous step status.

TRX files are written to subdirectories of the results folder, so use **/*.trx to find them recursively.

…tter in CI

Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

- Created helix_integration.proj for running integration tests on Helix - Created stage-helix-integration-tests.yml template for Helix integration - Updated ci.yml to use Helix for RunOniOS and RunOnAndroid categories - RunOniOS and RunOnAndroid now run on ARM64 macOS Helix queues only - Other integration test categories continue to run on Azure DevOps agents - Updated documentation to reflect Helix infrastructure Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

The Helix approach didn't work for integration tests because: - Integration tests need to build template projects from scratch - They need the MAUI workload installed - They require access to built packages from Pack stage - XUnitProject in Helix SDK doesn't support this complex setup Reverted to using standard Azure DevOps agents for RunOniOS and RunOnAndroid tests. macOS-15 hosted images are ARM64 (Apple Silicon) so the architecture fix still applies. Changes: - Removed eng/helix_integration.proj - Removed eng/pipelines/arcade/stage-helix-integration-tests.yml - Restored RunOniOS and RunOnAndroid in stage-integration-tests.yml - Updated documentation to reflect Azure DevOps infrastructure Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

- Added MacOSPoolArm64 (macOS-15, ARM64) and MacOSPoolX64 (macOS-13, x64) pool definitions - RunOnAndroid now runs only on ARM64 (macOS-15) - RunOniOS now runs on both ARM64 (macOS-15) and x64 (macOS-13) - Updated stage-integration-tests.yml to show vmImage in display names - Updated documentation to reflect dual-architecture iOS test lanes Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

- Set LogDirectory env var in setup-test-env.yml to match repoLogPath so binlogs are written to the artifact publish location - Fix DotnetInternal.cs to use TestEnvironment.GetLogDirectory() instead of non-existent TestEnvironment.LogDirectory property - Include project name in binlog filename for easier identification

- Reverted commits 55d2094 (xUnit conversion) and a2def73 (xUnit assertions) - Restored NUnit-based integration tests from before xUnit conversion - Kept important improvements: - XHarness known failure exit codes check (78-91) - SKIP_XCODE_VERSION_CHECK via TestConfig - Local package discovery paths - Device argument handling for XHarness - Deleted BuildTestFixture.cs (xUnit-specific, not needed for NUnit) - Updated instructions file to reference NUnit

Added comprehensive logging to help debug why log files aren't being published to Azure DevOps: - LogTestEnvironmentInfo() - logs test environment paths and env vars - LogTestDirectoryContents() - logs all files in test directory - LogLogDirectoryContents() - logs all files in log directory - Enhanced TearDown with detailed logging of attached files - Error handling with informative messages Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

- Split AppleTemplateTests into 7 individual test methods with specific categories (RunOniOS_MauiDebug, RunOniOS_MauiRelease, etc.) - Each iOS test now runs in its own CI job for faster debugging - Added CopyLogsToPublishDirectory() to copy logs to artifact location - Updated stage-integration-tests.yml to set IOS_TEST_DEVICE for all iOS tests - Updated ci.yml with individual test lanes (60 min timeout each) - Updated documentation to reflect new test structure Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

When building with PublishAot=true, NuGet tries to restore NativeAOT runtime packs for ALL target frameworks in the project, including Android. The Microsoft.NETCore.App.Runtime.NativeAOT.android-arm64/x64 packages aren't available in the configured NuGet sources. This fix restricts TargetFrameworks to net10.0-ios during NativeAOT builds to prevent attempting to restore Android NativeAOT packages.

The GetLogDirectory() function was: 1. Checking LogDirectory env var but then appending /logs to it 2. This caused logs to be written to the wrong directory Now when LogDirectory env var is set (from CI pipeline), it's used directly without appending /logs since it's already a complete log directory path. This ensures test binlogs and XHarness logs are written to the same directory that the pipeline publishes as artifacts.

The NativeAOT build was targeting ios-arm64 (device) instead of iossimulator-arm64 (simulator), causing the app to fail to launch on the simulator with APP_LAUNCH_TIMEOUT. Now passes TestEnvironment.IOSSimulatorRuntimeIdentifier to force the build to target the simulator architecture.

Move SkipXcodeVersionCheck and IosTestDevice properties from TestConfig.cs into TestEnvironment.cs to reduce indirection. Both classes serve the same purpose of reading environment configuration.

The maui-blazor template with TrimMode=full fails to build due to a known ASP.NET Core limitation. The Razor source generator produces code that accesses Router.NotFoundPage via reflection, triggering IL2111: error IL2111: Method 'Microsoft.AspNetCore.Components.Routing.Router.NotFoundPage.set' with parameters or return value with 'DynamicallyAccessedMembersAttribute' is accessed via reflection. Trimmer can't guarantee availability of the requirements of the method. Blazor doesn't fully support TrimMode=full for application assemblies. Tracking issue: dotnet/aspnetcore#63951 Docs: https://learn.microsoft.com/en-us/aspnet/core/blazor/host-and-deploy/configure-trimmer

Azure DevOps uppercases environment variable names when exposing them on macOS/Linux agents. The code was looking for 'LogDirectory' but the env var was set as 'LOGDIRECTORY'. Now checks for both cases to ensure log files are written to the correct artifact publish directory.

Refactored the integration test pipeline to use --filter "Name=TestMethodName" instead of requiring a separate Category constant for each individual test. Changes: - stage-integration-tests.yml now supports both testCategory (for groups) and testName (for individual tests) - ci.yml updated to use testName for RunOniOS_* tests - Removed individual RunOniOS_* category constants from Categories.cs - Simplified test attributes to only use the group category [Category(Categories.RunOniOS)] This reduces boilerplate when adding new tests - no longer need to add a category constant, just name the test method appropriately.

The test was pre-booting the simulator and passing the UDID to XHarness, but XHarness's watchdog disabling logic was shutting down and rebooting the simulator anyway. This caused 'simctl exit code 149' errors when the simulator was in an inconsistent state. Fix: Don't pre-boot the simulator. Just resolve the UDID in setup (to verify it can be found) and pass both target and UDID to XHarness. XHarness then handles the full simulator lifecycle (boot, install, run). Note: IOS_TEST_DEVICE must be set (e.g., 'ios-simulator-64_18.5') for XHarness to find the device. CI sets this env var in stage-integration-tests.yml.

The test was pre-booting the simulator and passing the UDID to XHarness, but XHarness's watchdog disabling logic was shutting down and rebooting the simulator anyway. This caused 'simctl exit code 149' errors when the simulator was in an inconsistent state. Fix: Don't pre-boot the simulator and don't pass a specific UDID. Let XHarness find the simulator based on --target (e.g., ios-simulator-64_18.5) and manage the entire lifecycle (find/create, boot, disable watchdogs, run). This requires IOS_TEST_DEVICE to include the iOS version (e.g., 'ios-simulator-64_18.5') so XHarness can find a matching simulator. CI sets this env var in the pipeline.

The test was pre-booting the simulator and passing the UDID to XHarness, but XHarness's watchdog disabling logic was shutting down and rebooting the simulator anyway. This caused 'simctl exit code 149' errors when the simulator was in an inconsistent state. Changes: - Don't pre-boot the simulator, let XHarness manage the entire lifecycle - Don't pass --device UDID, just --target (e.g., ios-simulator-64_18.5) - Auto-detect iOS version when IOS_TEST_DEVICE is not set or lacks version - Prefer iOS 18.x for XHarness compatibility (has iPhone XS device type) This makes tests work both on CI (with IOS_TEST_DEVICE=ios-simulator-64_18.5) and locally via Test Explorer (auto-detects available iOS runtime).

Azure DevOps uppercases environment variable names when exposing them on macOS/Linux agents. The code was looking for 'LogDirectory' but the env var was set as 'LOGDIRECTORY'. Now checks for both cases to ensure log files are written to the correct artifact publish directory.

- Simplify success detection: exit 0, 90 (timeout), or 83 with SIGKILL - Exit code 83 with 'mlaunch exited with 137' is the timeout kill, not a failure - Remove complex log pattern matching in favor of exit code checking - Add clear documentation of XHarness exit codes - Reduce default timeout from 75s to 15s - just need to verify app launches

The 75-second XHarness timeout was being consumed by simulator boot time (~30-35s on CI). By the time the simulator booted and started installing the app, there wasn't enough time left, causing premature cancellation. Fix: Pre-boot the simulator in SetUp so the full timeout is available for install + run.

Copilot AI assigned Copilot and PureWeen Dec 19, 2025

Copilot started work on behalf of PureWeen December 19, 2025 18:58 View session

Copilot AI changed the title ~~[WIP] Combine changes from PR 33168 and PR 33210~~ Combine integration test improvements from PRs #33168 and #33210 Dec 19, 2025

Copilot AI requested a review from PureWeen December 19, 2025 19:24

Copilot finished work on behalf of PureWeen December 19, 2025 19:24

Copilot started work on behalf of PureWeen December 19, 2025 20:39 View session

Copilot finished work on behalf of PureWeen December 19, 2025 20:47

Copilot started work on behalf of PureWeen December 19, 2025 23:05 View session

Copilot AI changed the title ~~Combine integration test improvements from PRs #33168 and #33210~~ Fix iOS integration tests on Apple Silicon and improve test infrastructure Dec 19, 2025

Copilot finished work on behalf of PureWeen December 19, 2025 23:13

Copilot started work on behalf of PureWeen December 21, 2025 21:11 View session

Copilot finished work on behalf of PureWeen December 21, 2025 21:22

Copilot started work on behalf of PureWeen December 21, 2025 21:31 View session

Copilot finished work on behalf of PureWeen December 21, 2025 21:42

Copilot started work on behalf of PureWeen December 21, 2025 22:14 View session

Copilot finished work on behalf of PureWeen December 21, 2025 22:22

Copilot started work on behalf of PureWeen December 21, 2025 23:43 View session

PureWeen and others added 17 commits December 23, 2025 14:57

- update global.json to use path

3e48aa7

Fix TRX publish path to use recursive glob pattern

5a07cbb

TRX files are written to subdirectories of the results folder, so use **/*.trx to find them recursively.

- update with proper instructions and tests

76f01b6

- additional updates so we can run locally and so everything works be…

fda152c

…tter in CI

-re-enable tests

836a99d

Convert integration tests from NUnit to xUnit

ec3681f

Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

Add descriptive error messages to xUnit assertions

c420f9c

Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>

- explicitly set arm pool

f45b997

PureWeen force-pushed the copilot/combine-pr-33168-and-33210 branch from 2ed13fe to 8b1dc3d Compare December 23, 2025 22:09

PureWeen added 12 commits December 23, 2025 16:28

Consolidate TestConfig into TestEnvironment

e456466

Move SkipXcodeVersionCheck and IosTestDevice properties from TestConfig.cs into TestEnvironment.cs to reduce indirection. Both classes serve the same purpose of reading environment configuration.

Revert SkipXcodeVersionCheck to false for CI

3b1547c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix iOS integration tests on Apple Silicon and add individual test lanes #33251

Fix iOS integration tests on Apple Silicon and add individual test lanes #33251

Copilot AI commented Dec 19, 2025 •

edited

Loading

Uh oh!

PureWeen commented Dec 19, 2025

Uh oh!

Copilot AI commented Dec 19, 2025

Uh oh!

PureWeen commented Dec 19, 2025

Uh oh!

Copilot AI commented Dec 19, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix iOS integration tests on Apple Silicon and add individual test lanes #33251

Are you sure you want to change the base?

Fix iOS integration tests on Apple Silicon and add individual test lanes #33251

Conversation

Copilot AI commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Architecture Detection

Individual iOS Test Lanes

CI Pipeline Fixes

File Location Logging & Log Publishing Fix

Uh oh!

PureWeen commented Dec 19, 2025

Uh oh!

Copilot AI commented Dec 19, 2025

Uh oh!

PureWeen commented Dec 19, 2025

Uh oh!

Copilot AI commented Dec 19, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

PureWeen commented Dec 21, 2025

Uh oh!

Copilot AI commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Dec 19, 2025 •

edited

Loading