Timeouts while pushing Sigstore logs to Rekor

[alex-berger]

Observed Behavior

After upgrading from podman 5.5.2 to 5.6.0 (on Alpine Linux) we observe an increased amount of failures (timeouts) when pushing Sigstore logs to Rekor. For example running commands like this:

podman --log-level debug image push --sign-by-sigstore ...

We now get errors like the one below:

Creating signature: Signing image using a sigstore signature
time="2025-08-20T11:11:45Z" level=debug msg="Calling Rekor's CreateLogEntry"
time="2025-08-20T11:11:45Z" level=debug msg="performing request" method=POST url="https://rekor.sigstore.dev/api/v1/log/entries"
Error: copying image 1/2 from manifest list: creating signature: Error uploading a log entry: context canceled
time="2025-08-20T11:11:46Z" level=debug msg="Shutting down engines"
time="2025-08-20T11:11:46Z" level=info msg="Received shutdown.Stop(), terminating!" PID=153

Expected Behavior

Uploading log entries must not timeout (unless there really is a availability issue with Rekor).

Workaround

Downgrade podmand to 5.5.2.

[Luap99]

We reworked the rekor code in #2873 so that likely caused it.
It would be nice to know what actually cancels the context here though I am not aware of way to get the info from go, @mtrmac do you know something?

[alex-berger]

@Luap99 Yeah, I am also suspecting #2873 or #2884 might have introduced this, but it could also be triggered by changes to podman (i.e. podman cancelling the Context).

However, I am completely lost (am not a Go expert) on how to narrow this down.

[mtrmac]

No, the context package does not record caller information.

Note that that’s not directly a timeout, context.DeadlineExceeded is formatted as context deadline exceeded. And from a quick check, I don’t see Podman having a timeout feature at all?!

---

Here’s a guess: would it be easy enough for you to remove these lines

image/signature/sigstore/rekor/openapi_infra.go

Lines 31 to 32 in ed9ae38

	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

, recompile Podman and re-test?

IIRC cancelling a context on a HTTP request before the body is read can terminate the request processing.

[Luap99]

IIRC cancelling a context on a HTTP request before the body is read can terminate the request processing.

Ah yes sounds like it

https://pkg.go.dev/net/http#NewRequestWithContext

For an outgoing client request, the context controls the entire lifetime of a request and its response: obtaining a connection, sending the request, and reading the response headers and body. See the Request type's documentation for the difference between inbound and outbound request fields.

So the cancel here is simply wrong then. Do we care about the cancel at all? We could just drop it I guess.

The local rekor server works fine in the unit test, I think the body is likely already populated on the local connection it only fails if the tcp connection was not fully read by the time we cancel and since that happens in different goroutines that races?

I was able to trigger the context canceled error with an example server like this

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"
)

func handler(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(201)
	flush := func() {
		if flusher, ok := w.(http.Flusher); ok {
			flusher.Flush()
		}
	}
	flush()
	time.Sleep(10 * time.Second)
	fmt.Fprintf(w, `{}`)
}

func main() {
	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":8080", nil))

[alex-berger]

This problem is still present in podman 5.7.0, @Luap99 can you please reopen this issue:

time="2026-01-09T15:54:42Z" level=debug msg="Uploading sigstore attachment manifest"
time="2026-01-09T15:54:42Z" level=debug msg="PUT https://<redacted>.dkr.ecr.eu-central-1.amazonaws.com/v2/nexiot-ag/ship/cicd/oci-images/alpine-ci-aws/manifests/sha256-5690652f6ce723fdda8e1d85ea9fb424211879b21b6c617c93a74b57196a08bf.sig"
Writing manifest list to image destination
time="2026-01-09T15:54:42Z" level=debug msg="Trying to use manifest list type application/vnd.oci.image.index.v1+json…"
time="2026-01-09T15:54:42Z" level=debug msg="PUT https://<redacted>.dkr.ecr.eu-central-1.amazonaws.com/v2/nexiot-ag/ship/cicd/oci-images/alpine-ci-aws/manifests/c871348d"
Creating signature: Signing image using a sigstore signature
time="2026-01-09T15:54:42Z" level=debug msg="Calling Rekor's CreateLogEntry"
time="2026-01-09T15:54:42Z" level=debug msg="performing request" method=POST url="https://rekor.sigstore.dev/api/v1/log/entries"
time="2026-01-09T15:54:43Z" level=fatal msg="creating signature: Error uploading a log entry: context canceled"

[mtrmac]

@alex-berger Podman v5.7.0 does contain the fix linked here, so if this still happens, I think it would be better tracked as a separate bug.