Operator keeps Terminating post Helm Install for Clickhouse chart #101
Description
HI Team,
When deploying ClickHouse using the altinity/clickhouse Helm chart (which bundles the operator), the ClickHouse Operator Deployment enters an infinite recreate/delete loop and the operator pod shuts down every ~10 seconds.
The operator logs repeatedly show graceful shutdown triggered by SIGTERM:
Shutdown signal received, waiting for all workers to finish
Stopping and waiting for caches
Stopping and waiting for webhooks
Wait completed, proceeding to shutdown the manager
export.go:83] Starting keeper OK
worker-boilerplate.go:42] shutdown request
Despite removing all CHI / CKI finalizers, the operator never becomes Ready and no operator pod stays Running.
Environment
Chart used: altinity/clickhouse (bundled operator)
Operator version: 0.25.5 (as deployed by chart)
Observed Behavior:
The operator Deployment never produces a running pod:
Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable
NewReplicaSet: <none>
OldReplicaSets: many (each with 0/0 pods)
Events show continuous scaling up/down of ReplicaSets:
Scaled up replica set XXXXX from 0 to 1
Scaled down replica set XXXXX from 1 to 0
(repeating endlessly)
Steps to reproduce:
Directly cloned and run helm install clickhouse . -n test -f values.yaml
operator pod logs:
I1128 13:29:22.442845 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 3 out of 11 I1128 13:29:22.442859 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 4 out of 11 I1128 13:29:22.442874 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 5 out of 11 I1128 13:29:22.442889 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 6 out of 11 I1128 13:29:22.442914 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 7 out of 11 I1128 13:29:22.442930 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 8 out of 11 I1128 13:29:22.442947 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 9 out of 11 I1128 13:29:22.442970 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 10 out of 11 I1128 13:29:22.442984 1 controller.go:583] Run():ClickHouseInstallation controller: starting worker 11 out of 11 I1128 13:29:22.443001 1 controller.go:593] Run():ClickHouseInstallation controller: workers started I1128 13:29:22.496591 1 controller.go:595] Run():ClickHouseInstallation controller: shutting down workers I1128 13:29:22.496622 1 thread_chi.go:107] export.go:68:E():end I1128 13:29:22.496648 1 worker-boilerplate.go:42] shutdown request 2025-11-28T13:29:22Z INFO Stopping and waiting for non leader election runnables 2025-11-28T13:29:22Z INFO shutting down server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"} 2025-11-28T13:29:22Z INFO Stopping and waiting for leader election runnables 2025-11-28T13:29:22Z INFO Shutdown signal received, waiting for all workers to finish {"controller": "clickhousekeeperinstallation", "controllerGroup": "clickhouse-keeper.altinity.com", "controllerKind": "ClickHouseKeeperInstallation"} 2025-11-28T13:29:22Z INFO All workers finished {"controller": "clickhousekeeperinstallation", "controllerGroup": "clickhouse-keeper.altinity.com", "controllerKind": "ClickHouseKeeperInstallation"} 2025-11-28T13:29:22Z INFO Stopping and waiting for caches 2025-11-28T13:29:22Z INFO Stopping and waiting for webhooks 2025-11-28T13:29:22Z INFO Wait completed, proceeding to shutdown the manager I1128 13:29:22.496996 1 export.go:83] Starting keeper OK You're using Lens Personal (for individuals or companies with < $10M annual revenue or funding)
deployment logs"
`kubectl describe deployment clickhouse-operator -n test
Name: clickhouse-operator
Namespace: test
CreationTimestamp: Fri, 28 Nov 2025 18:48:36 +0530
Labels: app.kubernetes.io/instance=clickhouse
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=operator
app.kubernetes.io/version=0.25.5
clickhouse.altinity.com/app=chop
clickhouse.altinity.com/chop=0.25.5
clickhouse.altinity.com/chop-commit=9ab22d8
clickhouse.altinity.com/chop-date=2025-10-24T08.40.12
helm.sh/chart=operator-0.25.5
Annotations: deployment.kubernetes.io/revision: 22
meta.helm.sh/release-name: clickhouse
meta.helm.sh/release-namespace: test
Selector: app.kubernetes.io/instance=clickhouse,app.kubernetes.io/name=operator
Replicas: 1 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: app.kubernetes.io/instance=clickhouse
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=operator
app.kubernetes.io/version=0.25.5
helm.sh/chart=operator-0.25.5
Annotations: checksum/confd-files: 16ee7854d6655b9ebeab0a8eb27348d690b6ba434373d0d7cfef349969e16551
checksum/configd-files: 89b144cc20e81d7a41a204b19ef322e78ce8faf21f7f0147caee52a71df6fe50
checksum/files: 333f4ec07e0e8d0288a7d5e21c78a702eb149dc942dda5b2cec9164b3411fe1f
checksum/keeper-confd-files: 65a6e8289c446b4ec3cda971222eeb08af4556479e25b7f5ad1acd9d12d2ddd1
checksum/keeper-configd-files: 75bd6a5fd9af7240357e59703ff4a8b35edad6691747fbb210968464c76575f9
checksum/keeper-templatesd-files: c411e1fe03cd1a5cddd7b71453117985173324ea7483f400739afe65d2611648
checksum/keeper-usersd-files: f80b64c532df80e8083563f2a8e6edbf7b761bb9fd08e7c2abbef86c74e82b36
checksum/templatesd-files: 1ef405e75285f6c316a0b078655b3219b2542a2c4742397158e87752530755d1
checksum/usersd-files: 6cc7a158960896e5701063cd368ba7f3e90dca9a84c1701a8483fe5f90f17ba6
clickhouse-operator-metrics/port: 9999
clickhouse-operator-metrics/scrape: true
prometheus.io/port: 8888
prometheus.io/scrape: true
Service Account: clickhouse-operator
Containers:
operator:
Image: altinity/clickhouse-operator:0.25.5
Port: 9999/TCP
Host Port: 0/TCP
Limits:
cpu: 150m
memory: 384Mi
Requests:
cpu: 100m
memory: 256Mi
Environment:
OPERATOR_POD_NODE_NAME: (v1:spec.nodeName)
OPERATOR_POD_NAME: (v1:metadata.name)
OPERATOR_POD_NAMESPACE: (v1:metadata.namespace)
OPERATOR_POD_IP: (v1:status.podIP)
OPERATOR_POD_SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
OPERATOR_CONTAINER_CPU_REQUEST: 1 (requests.cpu)
OPERATOR_CONTAINER_CPU_LIMIT: 1 (limits.cpu)
OPERATOR_CONTAINER_MEM_REQUEST: 268435456 (requests.memory)
OPERATOR_CONTAINER_MEM_LIMIT: 402653184 (limits.memory)
Mounts:
/etc/clickhouse-operator from etc-clickhouse-operator-folder (rw)
/etc/clickhouse-operator/chi/conf.d from etc-clickhouse-operator-confd-folder (rw)
/etc/clickhouse-operator/chi/config.d from etc-clickhouse-operator-configd-folder (rw)
/etc/clickhouse-operator/chi/templates.d from etc-clickhouse-operator-templatesd-folder (rw)
/etc/clickhouse-operator/chi/users.d from etc-clickhouse-operator-usersd-folder (rw)
/etc/clickhouse-operator/chk/conf.d from etc-keeper-operator-confd-folder (rw)
/etc/clickhouse-operator/chk/keeper_config.d from etc-keeper-operator-configd-folder (rw)
/etc/clickhouse-operator/chk/templates.d from etc-keeper-operator-templatesd-folder (rw)
/etc/clickhouse-operator/chk/users.d from etc-keeper-operator-usersd-folder (rw)
metrics-exporter:
Image: altinity/metrics-exporter:0.25.5
Port: 8888/TCP
Host Port: 0/TCP
Limits:
cpu: 150m
memory: 384Mi
Requests:
cpu: 100m
memory: 256Mi
Environment:
OPERATOR_POD_NODE_NAME: (v1:spec.nodeName)
OPERATOR_POD_NAME: (v1:metadata.name)
OPERATOR_POD_NAMESPACE: (v1:metadata.namespace)
OPERATOR_POD_IP: (v1:status.podIP)
OPERATOR_POD_SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
OPERATOR_CONTAINER_CPU_REQUEST: 1 (requests.cpu)
OPERATOR_CONTAINER_CPU_LIMIT: 1 (limits.cpu)
OPERATOR_CONTAINER_MEM_REQUEST: 268435456 (requests.memory)
OPERATOR_CONTAINER_MEM_LIMIT: 402653184 (limits.memory)
Mounts:
/etc/clickhouse-operator from etc-clickhouse-operator-folder (rw)
/etc/clickhouse-operator/chi/conf.d from etc-clickhouse-operator-confd-folder (rw)
/etc/clickhouse-operator/chi/config.d from etc-clickhouse-operator-configd-folder (rw)
/etc/clickhouse-operator/chi/templates.d from etc-clickhouse-operator-templatesd-folder (rw)
/etc/clickhouse-operator/chi/users.d from etc-clickhouse-operator-usersd-folder (rw)
/etc/clickhouse-operator/chk/conf.d from etc-keeper-operator-confd-folder (rw)
/etc/clickhouse-operator/chk/keeper_config.d from etc-keeper-operator-configd-folder (rw)
/etc/clickhouse-operator/chk/templates.d from etc-keeper-operator-templatesd-folder (rw)
/etc/clickhouse-operator/chk/users.d from etc-keeper-operator-usersd-folder (rw)
Volumes:
etc-clickhouse-operator-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-files
Optional: false
etc-clickhouse-operator-confd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-confd-files
Optional: false
etc-clickhouse-operator-configd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-configd-files
Optional: false
etc-clickhouse-operator-templatesd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-templatesd-files
Optional: false
etc-clickhouse-operator-usersd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-usersd-files
Optional: false
etc-keeper-operator-confd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-keeper-confd-files
Optional: false
etc-keeper-operator-configd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-keeper-configd-files
Optional: false
etc-keeper-operator-templatesd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-keeper-templatesd-files
Optional: false
etc-keeper-operator-usersd-folder:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: clickhouse-operator-keeper-usersd-files
Optional: false
Node-Selectors:
Tolerations:
Conditions:
Type Status Reason
Progressing True NewReplicaSetAvailable
Available False MinimumReplicasUnavailable
OldReplicaSets: clickhouse-operator-59b546bb94 (0/0 replicas created), clickhouse-operator-5fbb48bb97 (0/0 replicas created), clickhouse-operator-7ff7d8664 (0/0 replicas created), clickhouse-operator-d48848667 (0/0 replicas created), clickhouse-operator-7976c4f4db (0/0 replicas created), clickhouse-operator-548f5c58df (0/0 replicas created), clickhouse-operator-7ccc8cb796 (0/0 replicas created), clickhouse-operator-85d68fb799 (0/0 replicas created), clickhouse-operator-6b59d77f66 (0/0 replicas created), clickhouse-operator-7464d97f69 (0/0 replicas created), clickhouse-operator-58985f58f6 (0/0 replicas created)
NewReplicaSet:
Events:
Type Reason Age From Message
Normal ScalingReplicaSet 12m deployment-controller Scaled up replica set clickhouse-operator-767b59c645 from 0 to 1
Normal ScalingReplicaSet 12m deployment-controller Scaled down replica set clickhouse-operator-767b59c645 from 1 to 0
Normal ScalingReplicaSet 12m deployment-controller Scaled up replica set clickhouse-operator-74c86c5948 from 0 to 1
Normal ScalingReplicaSet 12m deployment-controller Scaled down replica set clickhouse-operator-74c86c5948 from 1 to 0
Normal ScalingReplicaSet 11m deployment-controller Scaled up replica set clickhouse-operator-78cf6f594c from 0 to 1
Normal ScalingReplicaSet 11m deployment-controller Scaled down replica set clickhouse-operator-78cf6f594c from 1 to 0
Normal ScalingReplicaSet 11m deployment-controller Scaled up replica set clickhouse-operator-5f56b6f4c8 from 0 to 1
Normal ScalingReplicaSet 11m deployment-controller Scaled down replica set clickhouse-operator-5f56b6f4c8 from 1 to 0
Normal ScalingReplicaSet 10m deployment-controller Scaled up replica set clickhouse-operator-675cb8f4cc from 0 to 1
Normal ScalingReplicaSet 2m53s (x26 over 10m) deployment-controller (combined from similar events): Scaled up replica set clickhouse-operator-7ccc8cb796 from 0 to 1
`
please suggest if any other ALtinity Helm for clickhouse is there or nay suggestions here please Thanks
Actually we need to deploy Clickhouse with 2 shards 2 replicas and create Opentelemetry based Raw tables and Materlised views to route tarcing data and to all query processing from grafana datasource will need more performance hence suggest your views please? and Also mature Clickhouse helm/operator. Thanks