Implement module extension for compilation tests [POC] by mateusz-olczyk · Pull Request #190 · EngFlow/gazelle_cc

mateusz-olczyk · 2026-02-02T13:17:00Z

A solution proposal for #189

This is a POC, lacking documentation, etc. I'll split it into logical parts if you accept this. Currently, only 2 test scenarios are included as examples:

subdirectory_basic
virtual_include_paths

How does it work?

Module generation

The module extension creates a new module for each test workspace. In the first phase, the contents of the test workspace are recursively copied into the new module. Actually, most of the files are symlinked for efficiency. BUILD.out files are renamed into BUILD.bazel files, so Bazel can interpret them and recognize packages. At this moment, we can already test the compilation with:

bazel build @generated_repo//...

Target aggregation

It would be great to include those targets in the command:

bazel build //...

This would create an effective feedback loop for the developer, allowing for an instant check for the success of a test scenario while working on a feature. But those targets belong to external, just-generated repositories. We need to aggregate them somehow.

Wildcards like ..., :all, :* are not allowed outside the CLI, i.e., in the Starlark code. We need to manually aggregate the targets within filegroups.

Filegroups generation

Now comes the tricky part. generate_filegroups.sh scans BUILD files using very primitive pattern matching and appends filegroups, so eventually we end up with labels like @generated_repo//:all_workspace_rules. With the help of gazelle_compilation_test macro, we include these into the root repository, so they become a part of //... wildcard and they're built together with gazelle_cc targets. So, actually gazelle_compilation_test does not define a Bazel test (it is just an alias for an external filegroup), but semantically we can call it a test.

I've decided to use a simple shell script because of limitation of the repository_rule. At this phase, Bazel targets do not even exist. So we cannot just use a go_binary as a tool. A shell script is simple; it depends only on basic commands like find or grep.

We could involve more correct parsing logic, similar to BUILD files discovery of gazelle. But it requires compiling a Go code manually, outside of Bazel rules logic. This is exactly how gazelle is built in bazel-gazelle (see go_repository_tools rule).

Summary

gazelle_compilation_tests module extension discovers the test workspaces under the given directory and includes them in the //... wildcard, giving the developer instant feedback about the compilation success.

No bazel_integration_test is used anywhere.

Signed-off-by: Mateusz Olczyk <molczyk@virtuslab.com>

jayconrod · 2026-02-02T18:46:47Z

bazel/gazelle_compilation_tests.bzl

+            doc = "Files with this suffix will be renamed to .bazel in the repository",
+        ),
+    },
+    local = True,


Is local absolutely needed? I think this causes the repo rule to be re-evaluated on every build, whether anything has changed or not.

According to repository_rule docs:

Indicate that this rule fetches everything from the local system and should be reevaluated at every fetch.

I had a problem with that, because the first statement: rule fetches everything from the local system is true, while I'm not sure about the second: should be reevaluated at every fetch. I'll experiment with turning this flag off and see if the repository_rule correctly reloads needed data when modifying test scenarios. Especially, I explicitly call watch, so it should do the job.

I haven't actually used one of these rules before, so I'm not really sure of the details. I vaguely thought they were for auto-configuring non-hermetic toolchains. For example, you could check the Xcode version and write that to a file that other targets depend on, so they would be rebuilt if you updated Xcode. (Not sure how accurate this impression is though)

jayconrod · 2026-02-02T18:51:24Z

bazel/gazelle_compilation_tests.bzl

+
+def _compilation_test_repo_impl(repository_ctx):
+    source_dir = repository_ctx.attr.source_dir
+    filegroup_tool = repository_ctx.path(Label("@gazelle_cc//:scripts/generate_filegroups.sh"))


This should be an attribute, rather than a dynamically generated label.

jayconrod · 2026-02-02T18:57:19Z

bazel/gazelle_compilation_tests.bzl

+            repository_ctx.symlink(source_file, dest_path)
+
+    # Generate filegroups
+    result = repository_ctx.execute([filegroup_tool, "-m", MAIN_FILEGROUP_NAME, "."] + repository_ctx.attr.rule_kinds)


It looks like we're running a pretty substantial script here.

In general, it is a good idea to put all commands in a script rather than trying to execute commands from Starlark. Can we move the commands above into the script so the logic isn't so fragmented?

Use caution in general though: Bash probably won't work on Windows, so we're going to need to copy and translate into Powershell, which is a pain. Keep this as minimal as possible.

In general, it is a good idea to put all commands in a script rather than trying to execute commands from Starlark.

Bash probably won't work on Windows

I'll try to do the opposite at first - maybe I'd be able to prepare a pure Starlark implementation. So no shells would be required at all.

jayconrod · 2026-02-02T18:58:24Z

bazel/gazelle_compilation_tests.bzl

+
+    return ctx.execute(find_command)
+
+def _compilation_test_repo_impl(repository_ctx):


Need some explanation as a doc string here or on the rule about what this does and why. I see that it's listing files and doing something with a filegroup, but the outcome is not at all clear from this context.

jayconrod · 2026-02-02T19:00:38Z

scripts/generate_filegroups.sh

@@ -0,0 +1,153 @@
+#!/usr/bin/env sh
+set -eu


More explicit and check for pipefail:

set -o errexit -o nounset -o pipefail

jayconrod · 2026-02-02T19:10:06Z

scripts/generate_filegroups.sh

So I see that this is creating filegroup targets. It feels heavy to me though. I wonder if it's better to just write a filegroup manually for each test? I'm mainly concerned that:

It's going to be a pain to duplicate and maintain this in Powershell for Windows.

This modifies the golden output files (I think?) in a way that's not obvious to the test writer.

I understand your concerns. But all this magic with filegroups has a deep sense from my point of view. This module extension should provide kind of a "framework" which allows easy extending already existing scenarios with a compilation checking. So, in theory, for valid scenarios, you'd be expected just to transparently include them in the new framework without any additional changes needed, especially in the test data content. If some changes are required, they should indicate that the scenario contained some real compilation errors from the very beginning, not because it is not adapted to the new test framework.

In my opionion it's up to the test framework to automatically provide some "boilerplate" to make the test runnable. Limiting the responsibility of the module extension only to a simple copying of contents with ".out"-".bazel" renaming seems to be incomplete. It's like half-automation. Some work is done for you, but in the end, you're forced to do something with your test layout anyway.

That's fair. I guess most of my concern is that it's written in Bash and wouldn't be compatible with Windows. If it's possible to do it in Starlark, that would be fine, though Starlark is a very limited language. A self-contained Python script might be a good alternative. We could reasonably expect a Python interpreter to be installed in CI.

mateusz-olczyk

Thank you, @jayconrod, for all the comments. I'll see what I can do.

mateusz-olczyk · 2026-02-03T12:43:33Z

scripts/generate_filegroups.sh

I understand your concerns. But all this magic with filegroups has a deep sense from my point of view. This module extension should provide kind of a "framework" which allows easy extending already existing scenarios with a compilation checking. So, in theory, for valid scenarios, you'd be expected just to transparently include them in the new framework without any additional changes needed, especially in the test data content. If some changes are required, they should indicate that the scenario contained some real compilation errors from the very beginning, not because it is not adapted to the new test framework.

In my opionion it's up to the test framework to automatically provide some "boilerplate" to make the test runnable. Limiting the responsibility of the module extension only to a simple copying of contents with ".out"-".bazel" renaming seems to be incomplete. It's like half-automation. Some work is done for you, but in the end, you're forced to do something with your test layout anyway.

mateusz-olczyk · 2026-02-03T12:45:32Z

bazel/gazelle_compilation_tests.bzl

+            repository_ctx.symlink(source_file, dest_path)
+
+    # Generate filegroups
+    result = repository_ctx.execute([filegroup_tool, "-m", MAIN_FILEGROUP_NAME, "."] + repository_ctx.attr.rule_kinds)


In general, it is a good idea to put all commands in a script rather than trying to execute commands from Starlark.

Bash probably won't work on Windows

I'll try to do the opposite at first - maybe I'd be able to prepare a pure Starlark implementation. So no shells would be required at all.

mateusz-olczyk · 2026-02-03T12:51:09Z

bazel/gazelle_compilation_tests.bzl

+            doc = "Files with this suffix will be renamed to .bazel in the repository",
+        ),
+    },
+    local = True,


According to repository_rule docs:

Indicate that this rule fetches everything from the local system and should be reevaluated at every fetch.

I had a problem with that, because the first statement: rule fetches everything from the local system is true, while I'm not sure about the second: should be reevaluated at every fetch. I'll experiment with turning this flag off and see if the repository_rule correctly reloads needed data when modifying test scenarios. Especially, I explicitly call watch, so it should do the job.

mateusz-olczyk added 2 commits February 2, 2026 12:10

Implement module extension for compilation tests [POC]

0a5f9c0

Signed-off-by: Mateusz Olczyk <molczyk@virtuslab.com>

Fix sh script

38140d3

Signed-off-by: Mateusz Olczyk <molczyk@virtuslab.com>

mateusz-olczyk requested review from WojciechMazur and jayconrod February 2, 2026 13:40

Fix the the call of filegroup_tool

043065f

Signed-off-by: Mateusz Olczyk <molczyk@virtuslab.com>

jayconrod reviewed Feb 2, 2026

View reviewed changes

mateusz-olczyk commented Feb 3, 2026

View reviewed changes

This was referenced Feb 5, 2026

Pure Starlark parser for Bazel BUILD files #191

Closed

Implement prepare_test_workspace.py script #192

Open


		return ctx.execute(find_command)

		def _compilation_test_repo_impl(repository_ctx):

Conversation

mateusz-olczyk commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How does it work?

Module generation

Target aggregation

Filegroups generation

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateusz-olczyk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateusz-olczyk commented Feb 2, 2026 •

edited

Loading