feat: introduce XcodeProjMapper to map XcodeProj to XcodeGraph #87

ajkolean · 2024-12-16T03:14:13Z

Description:
This pull request adds new functionality to convert .xcworkspace and .xcodeproj files into a high-level graph model (XcodeGraph). It introduces:

XcodeProjToGraph Module
A domain-focused module responsible for mapping raw Xcode structures (projects, workspaces, schemes, targets, and dependencies) into a cohesive XcodeGraph representation. This includes:
- Mappers:
  - WorkspaceMapper: Maps .xcworkspace contents into a Workspace domain model, identifying referenced .xcodeproj files and shared schemes.
  - ProjectMapper & PackageMapper: Extracts projects and associated packages, attributes, and configuration settings, ensuring each project is represented consistently.
  - TargetMapper & SchemeMapper: Converts PBX target definitions and scheme configurations into Target and Scheme domain entities, capturing essential build, test, and run information.
  - DependencyMapper & FileDependencyMapper: Translates XcodeProj target dependencies, platform conditions, and file references into a structured graph of interdependent components.
  - BuildPhaseMapper & BuildRuleMapper: Interprets build phases and rules, relating source files, resources, and frameworks to their respective domain-level constructs.
  - SettingsMapper & ConfigurationMatcher: Analyzes build settings and configurations, normalizing them into a predictable model that supports further graph analysis or code generation steps.
- Extensions & Helpers:
  Additional utilities to bridge the gap between raw XcodeProj objects and the domain model. For example, AbsolutePath+Extensions helps resolve file references, and Platform+Extensions assists in correctly inferring the platform from PBXTarget attributes.
- ProcessRunner & LipoTool:
  A utility to run external processes (like lipo) and integrate their output—such as architecture detection—into the graph model. This allows for richer metadata about binary artifacts and supported architectures.
TestSupport Module
This module centralizes mocks, helpers, and utilities that streamline testing. By providing reusable mock objects (e.g., mock project providers, file structures, and build settings), it reduces boilerplate and fosters consistency across all test suites. Key highlights:
- Mocks Folder: A collection of ready-to-use mocks for projects, targets, schemes, environment variables, and more.
- FileCreator & Structure Utilities: Simplifies creating and handling temporary files or directory structures during tests, ensuring clean and isolated test scenarios.
XcodeProjToGraphTests
A comprehensive test suite validating each piece of the mapping logic. The tests are thoughtfully organized by domain:
- MapperTests:
  - Build, Dependency, Settings, Target, Workspace: Ensure that each mapper converts raw Xcode structures (phases, dependencies, settings, etc.) into domain-level models accurately.
  - Project & Package: Validate correct interpretation of project attributes, packages, and configuration lists.
  - Scheme & Graph: Confirm that schemes are parsed and integrated into the workspace and project graphs, and that the overall graph mapper works as intended.
- ParserTests (ProjectParserTests):
  Verifies that the parsing logic for .xcodeproj and .xcworkspace directories is robust and reliable, providing a consistent foundation for the mappers.
- ProcessTests (LipoToolTests, ProcessRunnerTests):
  Checks the integration with external tools (lipo) and process running capabilities, ensuring errors are handled gracefully and output is captured correctly.

Next Steps - Integration Tests:
While this PR focuses on unit-level validation, an upcoming PR will introduce integration tests that run end-to-end scenarios using real .xcworkspace and .xcodeproj fixtures. This will confirm that the entire pipeline—from reading disk-based Xcode structures to producing a final XcodeGraph—functions correctly in practice.

While this PR focuses on core functionality, the integration tests validating end-to-end scenarios (e.g., parsing .xcworkspace and .xcodeproj into XcodeGraph) are complete. However, due to the size of the test fixtures (15,000 lines), they were removed to reduce noise.

To review and run these tests, check out this commit (before they were removed) : 75bf656. This includes real-world project fixtures and ensures the entire pipeline functions as expected in practical scenarios.

The results for these are browsable here: https://github.com/tuist/XcodeGraph/blob/75bf656adcec3b5fe7f4c034f797961361f9f584/Tests/XcodeProjToGraphTests/IntegrationTests/Projects/IntegrationTests%2BcommandLineToolWithDynamicFramework.swift

Testing & Verification:

Locally:
- Run swift build to compile all modules.
- Run swift test to execute the full test suite. All tests should pass, confirming correct functionality.
CI Integration:
Once integrated with CI (e.g., GitHub Actions), these tests will run automatically on each PR or commit, ensuring ongoing stability and reliability.

No Breaking Changes:
This PR adds new modules and tests without altering existing functionality. It forms the foundation for further enhancements and refinements.

Documentation & Maintenance:
Code comments and some documentation are included to guide future developers. With a clear architecture, it will be easier to extend capabilities, add integration tests, and enhance performance or features over time.

Summary:
This PR significantly expands the repository’s capabilities by introducing XcodeProjToGraph and TestSupport modules, along with a robust test suite in XcodeProjToGraphTests. Together, they provide a structured path from raw Xcode project/workspace data to a domain-level XcodeGraph model. The next step will be to add integration tests to validate complex real-world scenarios, ensuring the tool’s practicality and reliability for future use cases.

fortmarek

First of all, thanks a lot @ajkolean 👏

This work is huge and will enable powerful workflows that the Swift community has not seen, yet.

I did a first pass, trying to mostly focus on the higher level structure of the codebase.

Some of the comments are applicable across the PR, so it'd be amazing if you could have that in mind as I didn't want to repeat myself too much. I tried to point out when I saw some pattern we should imho redo in other places as well, but I might not have done this perfectly.

Overall, I am very onboard with the structure of breaking down the mapping of XcodeProj into multiple layers of individual mappers that have a simple responsibility.

Some of the patterns I'd consider changing (and these are also captured in the individual review comments):

Prefer structs over classes
Minimize the public interface of the XcodeProjToGraph module to the bare minimum. This will help us keep this repository stable.
Don't overdo it with running things in parallel. Benchmark first and make asynchronous only parts of the code that are actually resource intensive.
Use our utility for running shell commands instead of having a new implementation: https://github.com/tuist/command
Utilities should usually be configured directly via methods instead of passing the configuration at the init time. Unless there's a clear win by configuration utilities through init, let's keep init reserved for dependency injection.
Individual mappers should not be accessing layers above. For example, a PBXBuildPhase mapper should not have a notion of PBXTarget.

Let me know if we can help you in any way to proceed. This is top on my priority list, so I'll try to provide feedback swiftly here as you go through the review.

Once higher-level concerns are resolved, I'll do another pass going deeper into the individual implementations as I skimmed through some of them during the initial pass.

This must have been a ton of work and we really appreciate it – this is open source at its best 💜

Package.swift

Sources/TestSupport/Mocks/MockBuildFilesAndPhases.swift

Sources/XcodeProjToGraph/Mappers/Settings/ConfigurationMatcher.swift

Sources/XcodeProjToGraph/Mappers/TargetAndSchemes/TargetMapper.swift

Sources/XcodeProjToGraph/ProcessRunner/Executable.swift

Sources/XcodeProjToGraph/ProcessRunner/Lipo/LipoArchsResult.swift

Tests/XcodeProjToGraphTests/MapperTests/Xcode/Build/BuildRuleMapperTests.swift

… improve structure

ajkolean · 2024-12-19T03:56:37Z

@fortmarek
Thanks for the detailed feedback! I’m guessing waking up to what probably looked like a 7,000-line horror show wasn’t exactly fun. 🙈 I really appreciate you diving into it so thoroughly and quickly, especially with everything else on your plate, it means a lot.

I’ve gone through and addressed almost everything: cleaned up the mappers, streamlined the test data, and tightened up public access. If there’s anything I missed or further tweaks you’d suggest, I’m all ears.

Thanks again for taking the time to go through this so thoughtfully. Your sharp eye and collaborative approach made this a great process. Looking forward to any more thoughts! 🙏

fortmarek

Some additional feedback. Mostly nits. Some bigger pieces of feedback from this round I'd like to call out:

When we run into unknown objects and scenarios, I'd lean into throwing instead of swallowing these objects as nil. That will make it easier for us to spot our blinds spots. Otherwise, the conversion might be lossier than one would expect and then it's gonna be hard to figure out why.
I'd consider removing ProjectParser and making GraphMapper as the public interface. To do that, we'll need to make a PR in XcodeProj to accept an AbsolutePath in its init
Let's also update XcodeProj to hold its own path – I'd like us to move away from the ProjectProvider and WorkspaceProvider helpers and use XcodeProj directly.

fortmarek · 2024-12-19T15:02:11Z

Sources/XcodeGraph/Graph/GraphTarget.swift

 #if DEBUG
    extension GraphTarget {
-        public static func test(
+        static func test(


We actually use these in tuist/tuist – this would be a breaking change. We could have a debate on whether this is a good pattern or not, but for the purposes of this PR, I'd keep these test extension in XcodeGraph as public.

A reminder on this one.

fortmarek · 2024-12-19T15:15:11Z

Sources/XcodeProjMapper/Extensions/AbsolutePath+Extensions.swift

+extension AbsolutePath {
+    /// Maps a path by its extension to a `TargetDependency` if applicable.
+    ///
+    /// - Parameter condition: Optional platform condition.
+    /// - Returns: A `TargetDependency` if the extension matches known dependency types.
+    func mapByExtension(condition: PlatformCondition?) -> TargetDependency? {


I'd move this to its own mapper that follows the structure of other mappers. Something akin to:

struct PathDependencyMapper: PathDependencyMapping { func map(_ pathDependency: AbsolutePath) -> TargetDependency? { ... } }

Sources/XcodeProjMapper/Extensions/XCWorkspaceDataFileRef+Extensions.swift

Sources/XcodeProjMapper/Parser/ProjectParser.swift

Sources/XcodeProjMapper/Mappers/Graph/GraphMapper.swift

Sources/XcodeProjMapper/Mappers/Targets/PBXTarget+PlatformInference.swift

Sources/XcodeProjMapper/Mappers/Targets/PBXTargetDependencyMapper.swift

fortmarek · 2024-12-19T17:27:05Z

Sources/XcodeProjMapper/Mappers/Targets/PBXTargetMapper.swift

+        )
+    }
+
+    // MARK: - Helpers


Let's mark the helpers as private

Tuist/ProjectDescriptionHelpers/Module.swift

Tests/XcodeProjMapperTests/TestData/XcodeProj+TestData.swift

Tests/XcodeProjMapperTests/TestData/XCConfigurationList+TestData.swift

fortmarek · 2024-12-19T17:57:38Z

Tests/XcodeProjMapperTests/TestData/PBXProject+TestData.swift

+    }
+}
+
+//


I assume this can be removed?

fortmarek · 2024-12-19T17:58:45Z

Tests/XcodeProjMapperTests/TestData/.swift

Can be removed?

fortmarek · 2024-12-19T18:03:11Z

Tests/XcodeProjMapperTests/Mocks/WorkspaceFixture.swift

+
+// MARK: - WorkspaceFixture
+
+/// Provides references to sample Xcode workspace fixtures used for integration testing.


Let's create XcodeProjMapperIntegrationTests

fortmarek · 2024-12-19T18:05:53Z

Tests/XcodeProjMapperTests/MapperTests/Workspace/XCWorkspaceMapperTests.swift

+@Suite
+struct XCWorkspaceMapperTests {
+    @Test("Maps workspace without any projects or schemes")
+    func testMap_NoProjectsOrSchemes() throws {


I'd consider following the // Given // When // Then comments convention that we have in tuist/tuist. It makes tests easier to parse.

Tests/XcodeProjMapperTests/MapperTests/Workspace/XCWorkspaceMapperTests.swift

Tests/XcodeProjMapperTests/MapperTests/Target/TargetDependencyExtensionsTests.swift

Tests/XcodeProjMapperTests/IntegrationTests/IntegrationTests.swift

Sources/XcodeProjMapper/Mappers/Workspace/XCWorkspaceMapper.swift

fortmarek · 2024-12-21T12:18:08Z

That was not intended.

We should be able to remove now the ProjectParser, ProjectProvider, and WorkspaceProvider now that the path PR in XcodeProj was merged: tuist/XcodeProj#892

ajkolean · 2025-01-04T04:28:56Z

Apologies for the delay. I was away due to the holidays. I will respond and update from the feedback within the next few days. I haven't forgotten about it.

fortmarek · 2025-01-05T13:01:35Z

Apologies for the delay. I was away due to the holidays. I will respond and update from the feedback within the next few days. I haven't forgotten about it.

No worries, I figured 🙂 Hope you were able to recharge over the holidays! Looking forward to updates on this, we are still very much excited about this work landing.

ajkolean · 2025-01-23T15:54:53Z

Hey @fortmarek! Thanks for all the thorough feedback. I’ve just pushed the latest changes that (hopefully) address the remaining points. Apologies if I missed anything; please let me know if you spot any lingering issues or have more suggestions. 😬

fortmarek

This is huge @ajkolean. I have just a couple of last nits after which we can finally merge this 🚀 I am very excited about it 🙂

fortmarek · 2025-01-24T17:47:25Z

README.md

+import XcodeGraphMapper
+
+let mapper: XcodeGraphMapping = XcodeGraphMapper()
+let path = try AbsolutePath(validating: "/path/to/MyProjectOrWorkspace")
+let graph = try await mapper.map(at: path)
+
+*// You now have a Graph containing projects, targets, packages, and dependencies.*
+*// Example: print all target names across all projects*
+for project in graph.projects.values {
+    for (targetName, _) in project.targets {
+        print("Found target: \(targetName)")
+    }
+}


Suggested change

import XcodeGraphMapper

let mapper: XcodeGraphMapping = XcodeGraphMapper()

let path = try AbsolutePath(validating: "/path/to/MyProjectOrWorkspace")

let graph = try await mapper.map(at: path)

*// You now have a Graph containing projects, targets, packages, and dependencies.*

*// Example: print all target names across all projects*

for project in graph.projects.values {

for (targetName, _) in project.targets {

print("Found target: \(targetName)")

}

}

```swift

import XcodeGraphMapper

let mapper: XcodeGraphMapping = XcodeGraphMapper()

let path = try AbsolutePath(validating: "/path/to/MyProjectOrWorkspace")

let graph = try await mapper.map(at: path)

// You now have a Graph containing projects, targets, packages, and dependencies.*

// Example: print all target names across all projects*

for project in graph.projects.values {

for (targetName, _) in project.targets {

print("Found target: \(targetName)")

}

}

fortmarek · 2025-01-24T17:48:18Z

README.md

 )
 ```
+
+## XcodeGraphMapper


We also should start generating a docc documentation and serve that at tuist.dev. I'll add a note to myself to do that.

fortmarek · 2025-01-24T17:56:42Z

Sources/XcodeGraph/Models/Plist.swift


 // MARK: - InfoPlist

+public struct infoPlist: Equatable, Codable, Sendable {


It feels a bit odd that we differentiate between InfoPlist and infoPlist with just a different casing.

Whoops, this was an oversight, I was trying to decide between wrapper struct or include configs as associated values. I removed this, good catch!

fortmarek · 2025-01-24T18:04:04Z

Sources/XcodeProjMapper/Extensions/XCWorkspace+Extensions.swift

+
+extension XCWorkspace {
+    /// A computed property that either returns the workspace’s `path`
+    /// or throws a `MissingWorkspacePathError` if it’s `nil`.


Suggested change

/// or throws a `MissingWorkspacePathError` if it’s `nil`.

We crash if that happens, so we can remove that part of the comment.

fortmarek · 2025-01-24T18:04:11Z

Sources/XcodeProjMapper/Extensions/XCWorkspace+Extensions.swift

+
+extension XcodeProj {
+    /// A computed property that either returns the project’s `path`
+    /// or throws a `MissingProjectPathError` if it’s `nil`.


Suggested change

/// or throws a `MissingProjectPathError` if it’s `nil`.

fortmarek · 2025-01-24T18:05:16Z

Sources/XcodeProjMapper/Mappers/Graph/XcodeGraphMapper.swift

+
+    // MARK: - Public API
+
+    public func map(at path: AbsolutePath) async throws -> Graph {


I'd put this function also in the XcodeGraphMapping protocol.

It is in the protocol. I updated the internal parameter name of the protocol to match. Must of missed it when updating and compiler doesn't care if internal ones match

ajkolean · 2025-01-24T18:42:33Z

A few notes/reminders on somethings:

I’ve removed integration and performance tests for now. These can easily be added in a follow-up PR. To keep this PR focused and manageable, I opted not to introduce the fixtures at this stage.
This PR extracted some Metadata Providers from the main repo. We’ll need to ensure the main repo is updated accordingly to reflect this change.

fortmarek · 2025-01-27T20:51:37Z

I’ve removed integration and performance tests for now. These can easily be added in a follow-up PR. To keep this PR focused and manageable, I opted not to introduce the fixtures at this stage.

I'm aware of that, that's fine. Performance testing can be done a bit more ad-hoc instead of having these automated. We have a fixture generator in the tuist/tuist repository to generate arbitrarily large Xcode projects.

This PR extracted some Metadata Providers from the main repo. We’ll need to ensure the main repo is updated accordingly to reflect this change.

Yes! Would you be up to take that work? If not, I could try to carve out some time for it as well.

fortmarek · 2025-01-27T20:52:51Z

@ajkolean would you have some time to look into the CI failures? Those look legit.

ajkolean · 2025-01-27T22:47:26Z

Yes! Would you be up to take that work? If not, I could try to carve out some time for it as well.

Sure I can update after this in.

@ajkolean would you have some time to look into the CI failures? Those look legit.

Updated. I had to bump the macOS version and swift version due to some of the dependencies:

Compiling for macOS 12.0, but module 'FileSystem' has a minimum deployment target of macOS 13.0
Dependencies could not be resolved because 'command' >= 0.2.1 contains incompatible tools version (5.10.0)

fortmarek · 2025-01-28T08:55:05Z

Updated. I had to bump the macOS version and swift version due to some of the dependencies:

That's fine. But looks like there are still some CI issues.

ajkolean · 2025-01-28T22:31:38Z

That's fine. But looks like there are still some CI issues.

Seems like at least the linux ones are due to inaccessible URL APIs. tuist/FileSystem#103

fortmarek · 2025-02-05T08:29:33Z

Great work @ajkolean 👏 Let's iterate on the blogpost to share the news that this is now available with the community 💯

ajkolean added 6 commits December 15, 2024 16:41

[XcodeProjToGraph] add XcodeProjToGraph and tests

6416836

lint files

3209d11

[ProjectDescriptionHelpers] add fixtures to resources

5b91296

[WorkspaceMapperTests] add unit tests

75bf656

[Integration Tests] remove

5e57dc7

[access modifiers] add missing access modifiers

dbd55de

ajkolean changed the title ~~Introduce XcodeProjToGraph and TestSupport Modules, Plus Extensive Unit Tests for Workspace/Project Mapping~~ Introduce XcodeProjToGraph and TestSupport Modules for Workspace/Project Mapping Dec 16, 2024

ajkolean added 2 commits December 16, 2024 19:19

Merge branch 'main' into xcode-proj-to-graph

c3a1b49

[swift tools version] 5.9

7599100

fortmarek reviewed Dec 17, 2024

View reviewed changes

ajkolean added 2 commits December 18, 2024 21:54

Address code review feedback: refine mappers, clean up test data, and…

381e87d

… improve structure

[XcodeProj] mainPBXProject extension

1c124d0

fortmarek reviewed Dec 19, 2024

View reviewed changes

ajkolean mentioned this pull request Dec 20, 2024

feat: add path to XcodeProj and XCWorkspace tuist/XcodeProj#892

Merged

5 tasks

Merge branch 'main' into xcode-proj-to-graph

47b2f8a

fortmarek reviewed Dec 20, 2024

View reviewed changes

Sources/XcodeProjMapper/Mappers/Workspace/XCWorkspaceMapper.swift Outdated Show resolved Hide resolved

fortmarek closed this in tuist/XcodeProj#892 Dec 21, 2024

fortmarek reopened this Dec 21, 2024

ajkolean added 7 commits January 8, 2025 14:06

Merge branch 'main' into xcode-proj-to-graph

d5dc860

refactor: add accidental public removal back

024a0d1

feat: Strengthen error handling for unknown objects

f5b3242

feat: Introduce PathDependencyMapper

0707e4d

refactor: Remove ProjectParser & unify usage via XcodeGraphMapper

b38266f

chore: Bump XcodeProj version 8.26.0

f1da9e3

feat: Add command runner

8728c59

ajkolean added 7 commits January 23, 2025 06:03

[MockProvider] remove

916fada

[metadata] integrate into mappers

c33cb8e

[Phases] fix access modifiers

1d4cd61

[Tests] fix flaky tests

a4d436a

[Lint] lint files

bf35b6d

[Tuist] fix manifest

8c8c78e

[Doc] update readme

7ea75fe

Merge branch 'main' into xcode-proj-to-graph

2f3938d

ajkolean requested a review from fortmarek January 23, 2025 16:48

fortmarek approved these changes Jan 24, 2025

View reviewed changes

[plis] remove dead code

f1532cb

[Version] Update swift version to 5.10

d5907ed

[Lint] lint errors

92ae747

Fix build on Linux

7e3d31d

fortmarek changed the title ~~Introduce XcodeProjToGraph and TestSupport Modules for Workspace/Project Mapping~~ feat: introduce XcodeProjToGraph and TestSupport Modules for Workspace/Project Mapping Feb 4, 2025

fortmarek changed the title ~~feat: introduce XcodeProjToGraph and TestSupport Modules for Workspace/Project Mapping~~ feat: introduce XcodeProjMapper to map XcodeProj to XcodeGraph Feb 4, 2025

fortmarek added 4 commits February 4, 2025 20:05

Remove Project.swift

1b5320a

Remove cache workflow

b6acbf1

Skip testing XcodeMetadataTests

d6505a5

Update FileSystem

1ebea55

fortmarek merged commit 527101c into tuist:main Feb 5, 2025
6 checks passed

fortmarek mentioned this pull request Feb 6, 2025

fix: make Graph test factory public #107

Merged

+                  }
+              }
+              //


		// MARK: - WorkspaceFixture

		/// Provides references to sample Xcode workspace fixtures used for integration testing.


		// MARK: - InfoPlist

		public struct infoPlist: Equatable, Codable, Sendable {


		// MARK: - Public API

		public func map(at path: AbsolutePath) async throws -> Graph {

feat: introduce XcodeProjMapper to map XcodeProj to XcodeGraph #87

feat: introduce XcodeProjMapper to map XcodeProj to XcodeGraph #87

Uh oh!

Conversation

ajkolean commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fortmarek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajkolean commented Dec 19, 2024

Uh oh!

fortmarek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fortmarek commented Dec 21, 2024

Uh oh!

ajkolean commented Jan 4, 2025

Uh oh!

fortmarek commented Jan 5, 2025

Uh oh!

ajkolean commented Jan 23, 2025

Uh oh!

fortmarek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajkolean commented Dec 16, 2024 •

edited

Loading