enforce cpu/memory ceiling limit for prow jobs

[upodroid]

Part of #34139
/hold for discussion

We don't use the same instance sizes between clouds. Right now, we use an 8core 64gb RAM instance on GCP (c4|c4a|c4d-highmem-8) and a 16core 128gb node on AWS(r5ad.4xlarge or r5ad.2xlarge).

The preferred node size we should adopt consistently between clouds is the latest 8 core 32GB instance that has local SSDs.

Our nodes are underutilised in memory, as you can see in Datadog.

While we determine how to inject small, medium, and large pod sizes via Kyverno (mutating webhooks), I'll enforce a new change that caps the pod size to 7 cores and 27 GB of RAM. The remaining half core and a 1G of RAM goes to any agents we run on the cluster.

c4(30GB) -> c4d(31GB) -> c4a(32GB)

Less than 1% of prow jobs(54 out of 5874) request more than 28GB of RAM, and most of them don't need it.

 .venv  mahamed  Mac  ~  Desktop  Git  test-infra   enforce-cpu-memory-ceiling  3✎  1+  1⚑  USAGE  $  go test ./config/tests/jobs/ | grep "must be at most"
    jobs_test.go:1171: ci-k8sio-image-promo: container '' resources.limits[memory] (40Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build-1-31: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build-1-32: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build-1-33: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build-1-34: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-build-1-35: container '' resources.limits[memory] (34Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-cross-canary: container '' resources.limits[memory] (41Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-e2e-gce-scale-correctness: container '' resources.limits[memory] (39Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-e2e-gce-scale-resource-size: container '' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-e2e-kops-aws-scale-amazonvpc-using-cl2: container '' resources.limits[memory] (48Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-e2e-kops-aws-small-scale-amazonvpc-using-cl2: container '' resources.limits[memory] (48Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-1-31: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-1-32: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-1-33: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-1-34: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-1-35: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-dependencies: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-kubernetes-unit-golang-tip: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: ci-test-infra-triage: container '' resources.limits[memory] (48Gi) must be at most 28Gi
    jobs_test.go:1171: post-k8sio-image-promo: container '' resources.limits[memory] (40Gi) must be at most 28Gi
    jobs_test.go:1171: pull-cluster-api-addon-provider-helm-capi-e2e: container '' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kops-build: container '' resources.limits[memory] (64Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kops-test: container '' resources.limits[memory] (64Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kops-verify-golangci-lint: container '' resources.limits[memory] (64Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kops-verify-govet: container '' resources.limits[memory] (64Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kops-verify-hashes: container '' resources.limits[memory] (64Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-cross: container '' resources.limits[memory] (41Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-dependencies: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-dependencies: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-dependencies: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-golang-tip: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-golang-tip: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-e2e-unit-golang-tip: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-typecheck: container 'main' resources.limits[memory] (32Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit: container '' resources.limits[memory] (43Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-canary: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi
    jobs_test.go:1171: pull-kubernetes-unit-go-compatibility: container '' resources.limits[memory] (36Gi) must be at most 28Gi

Examples of misconfigured sizings:

1. https://monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=kubernetes&var-repo=kubernetes&var-job=ci-kubernetes-unit&var-build=All&from=now-24h&to=now Requested 43Gi but never exceeds 10Gi
2. https://monitoring-eks.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=kubernetes&var-repo=kubernetes&var-job=pull-kubernetes-typecheck&var-build=All&from=now-2d&to=now requested 32Gi but never exceeds 10Gi

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• config/tests/jobs/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@upodroid: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-test-infra-unit-test	7955c13	link	true	/test pull-test-infra-unit-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.