feat: add support for preserving and labeling intermediate stage images #6556

ezopezo · 2025-12-01T14:16:45Z

This adds support for preserving and labeling intermediate stage images in multi-stage builds. In contrast to the --layers flag, --cache-stages preserves only the final image from each named stage (FROM ... AS name), not every instruction layer. This also keeps the final image's layer count unchanged compared to a regular build.

New flags:

--cache-stages: preserve intermediate stage images instead of removing them
--stage-labels: add metadata labels to intermediate stage images (stage name, base image, build ID, parent stage name). Requires --cache-stages.
--build-id-file: write unique build ID (UUID) to file for easier identification and grouping of intermediate images from a single build. Requires --stage-labels.

The implementation also includes:

Detection of transitive alias patterns (stage using another intermediate stage as base)
Validation that --stage-labels requires --cache-stages
Validation that --build-id-file requires --stage-labels
Test coverage (15 tests) and documentation updates

What type of PR is this?

/kind feature

What this PR does / why we need it:

General use: This functionality is useful for identification, debugging, and reusing intermediate stage images in multi-stage builds.

Specific need: Identifying the content copied from intermediate stages in multi-stage builds into the final image is a hard requirement for supporting Contextual SBOM - an SBOM that understands the origin of each component.
While intermediate images can be extracted using the --layers option, this approach has several issues for our use case:

Intermediate stage images are unlabeled, making it difficult to determine which image corresponds to which build stage - especially when the Containerfile reuses the same pullspec across multiple stages.
All instructions from all intermediate stages appear in the cache (visible via buildah images --all), which introduces unnecessary noise for our purposes.
rootfs.diff_ids are not squashed in final stage: the final-stage image ends up containing diff IDs for every instruction in the final stage. However, we need the final build image to resemble a regular build (without --layers), meaning:
- it should contain the diff IDs inherited from the base image, and
- exactly one diff ID representing the squashed final-stage instructions.

Related repositories:
konflux (uses mobster for SBOM generation),
mobster (implements contextual SBOM functionality requiring this change),
capo (wraps builder content identification functionality for mobster),
Contact person: emravec (RedHat) / @ezopezo (Github)

How to verify it

Run any multistage build with intermediate stage specified with implemented arguments. Resulting intermediate images should be correctly labeled. Example:
buildah build --cache-stages --stage-labels --build-id-file ./file.txt -t test:0.1 .

Which issue(s) this PR fixes:

Fixes: #6257
Internal Jira: https://issues.redhat.com/browse/ISV-6122

Does this PR introduce a user-facing change?

Add `--cache-stages`, `--stage-labels`, and `--build-id-file` flags for preserving and labeling intermediate stage images in multi-stage builds.

openshift-ci · 2025-12-01T14:16:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ezopezo
Once this PR has been reviewed and has the lgtm label, please assign luap99 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

packit-as-a-service · 2025-12-01T14:18:30Z

Ephemeral COPR build failed. @containers/packit-build please check.

ezopezo · 2025-12-02T21:17:22Z

/retest

openshift-ci · 2025-12-02T21:17:38Z

@ezopezo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ezopezo · 2025-12-02T21:38:11Z

@nalind can you please take a look and put ok-to-test label? It seems to me that tests are failing most likely with some timeouts and thus I would like to try to re-run them (or please tell me what I just broke :) ).

nalind · 2025-12-02T22:05:26Z

/ok-to-test

ezopezo · 2025-12-03T09:22:06Z

/test

openshift-ci · 2025-12-03T09:22:22Z

@ezopezo: No presubmit jobs available for containers/buildah@main

Details

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ezopezo · 2025-12-04T09:21:56Z

@nalind @mtrmac @TomSweeneyRedHat can you please take a look on this? (or pick up some appropriate reviewers?) Thanks in advance!

mtrmac · 2025-12-09T13:45:13Z

@flouthoc if you have time

TomSweeneyRedHat · 2026-01-06T15:30:54Z

@Luap99 @nalind PTAL

docs/buildah-build.1.md

TomSweeneyRedHat · 2026-01-08T00:19:41Z

docs/buildah-build.1.md


 buildah build --layers -t imageName .

+buildah build --cache-stages --stage-labels -t imageName .


just to show the bool values in play, perhaps

Suggested change

buildah build --cache-stages --stage-labels -t imageName .

buildah build --cache-stages false --stage-labels false -t imageName .

I skipped that since this is a relatively uncommon use case for boolean flags, because the false behavior is implicit whenever the flag is absent. Other flags such as --no-cache or --layers follow the same pattern (present when true, absent when not, without presence of the --layers false) so I wanted to stay consistent with your docs. If you feel strongly about it, I certainly can add the examples :)

There's a long-standing bit of confusion in the man pages where, even though the boolean argument values are optional, we don't suggest that they always have to be supplied in the --flag=value form, with an equal sign, to prevent them from being treated as unrelated arguments.

@nalind Does it make a sense to you to add args with equal sign and boolean or leave it as it is? What do you think?

TomSweeneyRedHat · 2026-01-08T00:20:51Z

docs/buildah-build.1.md

+Find an intermediate image for a specific stage name:
+
+buildah images --filter "label=io.buildah.stage.name=builder"
+


This is great! TY!

TomSweeneyRedHat · 2026-01-08T00:33:20Z

pkg/cli/build.go

 		LayerLabels:             iopts.LayerLabel,
 		Layers:                  layers,
+		CacheStages:             iopts.CacheStages,
+		StageLabels:             iopts.StageLabels,


Please add these three options in alphabetically. I.e. CacheStages at line 396.5 and so on.

TomSweeneyRedHat · 2026-01-08T00:34:35Z

pkg/cli/common.go

 	Format              string
 	From                string
 	Iidfile             string
+	BuildIDFile         string


To keep making my OCD happy, please move this to line 63.5

TomSweeneyRedHat · 2026-01-08T00:35:05Z

pkg/cli/common.go

-	Layers  bool
+	ForceRm     bool
+	Layers      bool
+	CacheStages bool


More OCD alpha changes, please put this at line 28.5

TomSweeneyRedHat · 2026-01-08T00:41:01Z

tests/bud.bats

+
+  run_buildah rmi --all
+}
+


can you update and/or add tests to use "true" and "false" for the boolean flags? Leave at least one using the default true for each too.

For false I added two new tests for --cache-stages and --stage-labels and for true I updated one existing test 👍

TomSweeneyRedHat

First pass LGTM, just a couple of nitty things here and there.

@nalind or @Luap99 PTAL

Also, have you tried vendoring Buildah into Podman with these changes yet?

@ddarrah should we have a feature/epic for this in RHEL 9.8/10.2?

nalind

Overall I think I understand the goals here. The new flags interact with --layers in confusing ways, though - the build ID recorded for --build-id-file won't match the image whose ID we return if the build is a cache hit, but we may quietly commit new versions of the final product of intermediate stages. I'm not really sure what the intended result is there.

imagebuildah/executor.go

nalind · 2026-01-13T15:59:49Z

define/build.go

+	CacheStages bool
+	// StageLabels tells the builder to add metadata labels to intermediate stage images for easier recognition.
+	// These labels include stage name, base image, build ID, and parent stage name (when a stage uses another
+	// intermediate stage as its base, i.e., transitive aliases). This option requires CacheStages to be enabled.


Even as it's being defined, "transitive aliases" doesn't make any sense to me as jargon. Does the term come from somewhere else?

Fair point - you're right that “transitive alias” isn't established jargon. I started using it internally as shorthand for multi-stage builds where one named stage is used as the base for another, and it unfortunately leaked here. I think I picked it up from an article or docs at some point, but I can't find a reference now - which indeed makes it sound like it was revealed to me in a dream 🙂
I'll happily rephrase all occurrences using some more standard terminology!

nalind · 2026-01-13T16:08:11Z

docs/buildah-build.1.md


 buildah build --layers -t imageName .

+buildah build --cache-stages --stage-labels -t imageName .


There's a long-standing bit of confusion in the man pages where, even though the boolean argument values are optional, we don't suggest that they always have to be supplied in the --flag=value form, with an equal sign, to prevent them from being treated as unrelated arguments.

nalind · 2026-01-13T18:17:32Z

imagebuildah/stage_executor.go

+			logrus.Debugf("Committing intermediate stage %s (index %d) for --cache-stages", s.name, s.index)
+			createdBy := fmt.Sprintf("/bin/sh -c #(nop) STAGE %s", s.name)
+			// Commit the stage without squashing, using empty output name (intermediate image)
+			imgID, commitResults, err = s.commit(ctx, createdBy, false, "", false, false)


Why set emptyLayer as false here?

You re right - it should not be false. I updated it such way, if stage was already committed, we only add metadata labels without creating a duplicate empty layer. Let me know, if it is okay!

Actually, I'm thinking that I was confused here - there are cases where we don't commit after the final instruction in a stage, since we're not going to base later stages off of it, and the working container is sufficient for COPY --from references to it later (for example, when imageIsUsedLater is false in the single-layer build case). I think we're okay with false for multi-layer builds, but should probably give it a once-over again.

My aim with emptyLayer := imgID != "" is to not duplicate an empty layer created by the previous commit that already occurred in the execute function.

When imgID != "" (this means to me that stage was already committed):

Regular builds (without –layers) commit the stage in execute function if imageIsUsedLater=true

Also now with --layers, every instruction is commited, so imgID is set
I think, in these cases, we only need to add labels by commit and emptyLayer=true should prevent creating a duplicate layer in the intermediate image (right?).

When imgID == "" (I grasp this as stage was not committed and is left as working container) regular builds with imageIsUsedLater=false don't commit the stage, but we need to capture filesystem changes of the working container- here is emptyLayer=false okay because ensures we create a proper layer with the stage's content..

I think this prevents duplicate layers while correctly capturing filesystem changes when the stage hasn't been committed yet (and we need them).
Let me know if there's an edge case I'm missing or I got tangled into this too much and I am wrong 😀

ezopezo · 2026-01-15T09:18:54Z

Overall I think I understand the goals here. The new flags interact with --layers in confusing ways, though - the build ID recorded for --build-id-file won't match the image whose ID we return if the build is a cache hit, but we may quietly commit new versions of the final product of intermediate stages. I'm not really sure what the intended result is there.

Does it make a sense to add an mutual exclusion condition that --layers and --cache-stages cannot be used together (as serve fundamentally different purposes)? I want to keep this functionality as isolated as possible from others to avoid unnecessary complexities like this.

ezopezo · 2026-01-20T15:27:18Z

@nalind Would you be able to take an another look please? Thanks in advance!

nalind · 2026-01-21T23:10:36Z

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it.
If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate.
Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

ezopezo · 2026-01-26T16:36:40Z

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it. If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate. Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

Yes, it works well buildah build --layers --cache-stages --stage-labels stores all the layers of the Containerfile, correctly labels them and subsequent plain buildah build --layers reuses those by hitting cache - those layers remain intact, labels are preserved. Are we okay with such behavior?

ezopezo · 2026-01-26T17:06:30Z

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman?
(I believe this is what @TomSweeneyRedHat asked before as well):

Also, have you tried vendoring Buildah into Podman with these changes yet?

nalind · 2026-01-26T22:37:48Z

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman? (I believe this is what @TomSweeneyRedHat asked before as well):

Since we're on main, check out podman's main branch, and use go mod edit -replace github.com/containers/buildah=github.com/ezopezo/buildah@478bf2159f6ffe878c5162ed1d83c69ae3e6a6d6 && go mod tidy && go mod vendor && make to build. (If your branch name was a valid version string, you could use its name directly instead of having to look up the commit ID.) At that point you can either attempt to test locally with make .install.ginkgo localintegration and related targets, or commit everything to a branch, push it to a podman fork, and open a PR to have podman's CI do it, taking care to mark it as not intended for merging.

This adds support for preserving and labeling intermediate stage images in multi-stage builds. In contrast to the --layers flag, --cache-stages preserves only the final image from each named stage (FROM ... AS name), not every instruction layer. This also keeps the final image's layer count unchanged compared to a regular build. New flags: - --cache-stages: preserve intermediate stage images instead of removing them - --stage-labels: add metadata labels to intermediate stage images (stage name, base image, build ID, parent stage name). Requires --cache-stages. - --build-id-file: write unique build ID (UUID) to file for easier identification and grouping of intermediate images from a single build. Requires --stage-labels. The implementation also includes: - Detection of transitive alias patterns (stage using another intermediate stage as base) - Validation that --stage-labels requires --cache-stages - Validation that --build-id-file requires --stage-labels - Test coverage (15 tests) and documentation updates This functionality is useful for debugging, exploring, and reusing intermediate stage images in multi-stage builds. Signed-off-by: Erik Mravec <emravec@redhat.com>

- removed unused buildIDFile - removed "transitive alias" wording - update logic of the emptyLayer arg in commit - --layers and --cache-stages are mutually exclusive This commit will be suqashed serves only for review purposes. Signed-off-by: Erik Mravec <emravec@redhat.com>

--layers and --cache-stages can be used together. When --cache-stages is enabled (with or without --layers) checkForLayers is set to false, blocking cache lookup (fresh build every time with --cache-stages). When --cache-stages is enabled with --layers together, all other subsequent builds without --cache-stages can use cached layers. This commit will be squashed serves only for review purposes. Signed-off-by: Erik Mravec <emravec@redhat.com>

ezopezo · 2026-01-28T11:05:40Z

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman? (I believe this is what @TomSweeneyRedHat asked before as well):

Since we're on main, check out podman's main branch, and use go mod edit -replace github.com/containers/buildah=github.com/ezopezo/buildah@478bf2159f6ffe878c5162ed1d83c69ae3e6a6d6 && go mod tidy && go mod vendor && make to build. (If your branch name was a valid version string, you could use its name directly instead of having to look up the commit ID.) At that point you can either attempt to test locally with make .install.ginkgo localintegration and related targets, or commit everything to a branch, push it to a podman fork, and open a PR to have podman's CI do it, taking care to mark it as not intended for merging.

@nalind Thank you I did it according your guide and created this testing PR. I think it is failing because of the docs mismatch and exceeding some limit of the binary size (which could be overridden by label I think) and also I think I need from maintainers approval for running all of the jobs, could you help me with this please?

openshift-ci bot added the kind/feature Categorizes issue or PR as related to a new feature. label Dec 1, 2025

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2025

ezopezo force-pushed the emravec/preserve-intermediate-images branch from 59cd9ae to 6bd3187 Compare December 1, 2025 14:24

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2025

ezopezo force-pushed the emravec/preserve-intermediate-images branch 2 times, most recently from b7e81df to 3314051 Compare December 2, 2025 16:30

openshift-ci bot added the ok-to-test label Dec 2, 2025

ezopezo force-pushed the emravec/preserve-intermediate-images branch from 3314051 to 3955b20 Compare December 3, 2025 09:25

TomSweeneyRedHat reviewed Jan 8, 2026

View reviewed changes

docs/buildah-build.1.md Outdated Show resolved Hide resolved

TomSweeneyRedHat reviewed Jan 8, 2026

View reviewed changes

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Jan 9, 2026

ezopezo force-pushed the emravec/preserve-intermediate-images branch 2 times, most recently from 6b2baab to 3272226 Compare January 13, 2026 16:07

nalind reviewed Jan 13, 2026

View reviewed changes

ezopezo force-pushed the emravec/preserve-intermediate-images branch 2 times, most recently from 0fee359 to a54b1ee Compare January 27, 2026 14:14

ezopezo mentioned this pull request Jan 27, 2026

draft: running tests DO NOT MERGE containers/podman#27964

Open

ezopezo force-pushed the emravec/preserve-intermediate-images branch from a54b1ee to 1bfc86d Compare January 27, 2026 16:31

ezopezo added 3 commits January 28, 2026 10:28

ezopezo force-pushed the emravec/preserve-intermediate-images branch from 1bfc86d to 89f7b79 Compare January 28, 2026 09:28


		buildah build --layers -t imageName .

		buildah build --cache-stages --stage-labels -t imageName .

	buildah build --cache-stages --stage-labels -t imageName .
	buildah build --cache-stages false --stage-labels false -t imageName .

		Find an intermediate image for a specific stage name:

		buildah images --filter "label=io.buildah.stage.name=builder"

feat: add support for preserving and labeling intermediate stage images #6556

Are you sure you want to change the base?

feat: add support for preserving and labeling intermediate stage images #6556

Uh oh!

Conversation

ezopezo commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

How to verify it

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Dec 1, 2025

Uh oh!

packit-as-a-service bot commented Dec 1, 2025

Uh oh!

ezopezo commented Dec 2, 2025

Uh oh!

openshift-ci bot commented Dec 2, 2025

Uh oh!

ezopezo commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nalind commented Dec 2, 2025

Uh oh!

ezopezo commented Dec 3, 2025

Uh oh!

openshift-ci bot commented Dec 3, 2025

Uh oh!

ezopezo commented Dec 4, 2025

Uh oh!

mtrmac commented Dec 9, 2025

Uh oh!

TomSweeneyRedHat commented Jan 6, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomSweeneyRedHat left a comment

Choose a reason for hiding this comment

Uh oh!

nalind left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ezopezo commented Dec 1, 2025 •

edited

Loading

ezopezo commented Dec 2, 2025 •

edited

Loading

ezopezo commented Jan 26, 2026 •

edited

Loading