OCPBUGS-64847: UPSTREAM: 135298: kubelet: remove rejected pods from podManager to fix resource accounting

[sjenning]

/kind bug

Upstream pick kubernetes#135296 to address occasional failures in conformance and an actual regression in the kubelet.

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330970178883584

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330967670689792

Manifests with OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000613, capacity: 3500

[openshift-ci-robot]

@sjenning: This pull request references Jira Issue OCPBUGS-69714, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/kind bug

Upstream pick kubernetes#135296 to address occasional failures in conformance.

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330970178883584

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330967670689792

Manifests with OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000613, capacity: 3500

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@sjenning: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 1c66f5b|UPSTREAM: 135298: kubelet: remove rejected pods from podManager to fix resource accounting: the upstream PR kubernetes/kubernetes#135298 has not yet merged

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[sjenning]

/jira refresh

[openshift-ci-robot]

@sjenning: This pull request references Jira Issue OCPBUGS-69714, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lyman9966

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sjenning]

cc @harche @haircommander

[openshift-ci-robot]

@sjenning: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

/kind bug

Upstream pick kubernetes#135296 to address occasional failures in conformance.

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330970178883584

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330967670689792

Manifests with OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000613, capacity: 3500

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@sjenning: This pull request references Jira Issue OCPBUGS-64847, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/kind bug

Upstream pick kubernetes#135296 to address occasional failures in conformance.

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330970178883584

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330967670689792

Manifests with OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000613, capacity: 3500

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sjenning]

/jira refresh

[openshift-ci-robot]

@sjenning: This pull request references Jira Issue OCPBUGS-64847, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sjenning]

/payload-aggregate periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance 10

[openshift-ci]

@sjenning: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8c125810-dba3-11f0-8366-fa03ceaee7b2-0

[haircommander]

from the looks of payload jobs, only two jobs failed that test (1 2) and neither were with the error this is attempting to fix
/retest
/lgtm

while upstream hasn't merged yet, it seems to be getting good feedback, and there's a unit test that clearly shows this is the error

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/kubelet/DOWNSTREAM_OWNERS [sjenning]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[haircommander]

/cc @bertinatto
/verified by [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory limits only @haircommander

[openshift-ci-robot]

@haircommander: This PR has been marked as verified by [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] Burstable QoS pod,one container with cpu & memory requests + limits - increase memory limits only @haircommander.

Details

In response to this:

/cc @bertinatto
/verified by [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory limits only @haircommander

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@sjenning: This pull request references Jira Issue OCPBUGS-64847, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/kind bug

Upstream pick kubernetes#135296 to address occasional failures in conformance and an actual regression in the kubelet.

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330970178883584

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance/2001330967670689792

Manifests with OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000613, capacity: 3500

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@sjenning: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[rphillips]

/remove-label backports/unvalidated-commits

[rphillips]

/hold

[openshift-ci]

@rphillips: Can not set label backports/unvalidated-commits: Must be member in one of these teams: [openshift-staff-engineers]

Details

In response to this:

/remove-label backports/unvalidated-commits

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rphillips]

/hold cancel

[sjenning]

/payload-aggregate periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance 10

[openshift-ci]

@sjenning: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/63e89980-dc39-11f0-95ef-b9b919db99ef-0

[wking]

One of the new aggregation's runs also had a test-case fail on on the poisoned resource-cache behavior:

: [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory limits only expand_less	3s
{  fail [k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1062]: Told to stop trying after 2.007s.
The phase of Pod resize-test-b6tdn is Failed which is unexpected.}

with stdout for that test-case mentioning:

I1218 19:04:57.141743 2378 dump.go:53] At 2025-12-18 19:04:55 +0000 UTC - event for resize-test-b6tdn: {kubelet ip-10-0-11-57.ec2.internal} OutOfcpu: Node didn't have enough resource: cpu, requested: 20, used: 1000000000000688, capacity: 3500

We have resources to help debug this time, including HyperShift dump artifacts, but if there are Node logs or other assets with kubelet versions to confirm this change is included in the ip-10-0-11-57.ec2.internal kubelet, I haven't found those yet. Maybe someone with more familiarity with HyperShift CI artifact structure can find those logs?

[haircommander]

hm I tried to poke around through the logs but can't find any kubelet logs for any of the hosted workers. @sjenning is that expected

[openshift-bot]

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

[openshift-ci-robot]

@openshift-bot: This pull request references Jira Issue OCPBUGS-64847, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[harche]

alternate approach, #2553 in light of kubernetes#135298 (comment)