CockroachDB panics on rack3

[augustuswm]

While preparing rack3 for the R17 and R17.1 updates we detected core files from crdb.

Log files from near the time of the core file look like:

E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191  a panic has occurred!
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +pebble/table: invalid table 14889595 (checksum mismatch at 4168591/548677)
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +(1) attached stack trace
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  -- stack trace:
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | runtime.gopanic
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /opt/ooce/go-1.22/src/runtime/panic.go:770
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:251
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | runtime.gopanic
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /opt/ooce/go-1.22/src/runtime/panic.go:770
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_send.go:95
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | runtime.gopanic
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /opt/ooce/go-1.22/src/runtime/panic.go:770
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).destroy
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/pebble_iterator.go:810
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Close
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/pebble_batch.go:111
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_proposal.go:740
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_proposal.go:829
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_raft.go:113
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_write.go:167
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_send.go:460
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_send.go:156
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_send.go:197
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:191
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/server/node.go:1011
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/server/node.go:994
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/server/node.go:1063
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9495
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.ServerInterceptor.func13
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:137
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1.1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1117
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func3
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/rpc/context.go:238
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1.1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1120
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1.1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/rpc/context.go:207
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/rpc/context.go:205
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1.1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1120
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1122
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9497
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | google.golang.org/grpc.(*Server).processUnaryRPC
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1283
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +Wraps: (2) forced error mark
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | ‹"pebble: corruption"›
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +Wraps: (3) attached stack trace
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  -- stack trace:
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/internal/base/error.go:27
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/sstable.checkChecksum
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/sstable/reader.go:2281
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/sstable.(*Reader).readBlock
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/sstable/reader.go:2337
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/sstable.(*Reader).readFilter
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/sstable/reader.go:2250
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).seekPrefixGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/sstable/reader.go:633
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).SeekPrefixGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/sstable/reader.go:616
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/level_iter.go:434
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble.(*mergingIter).seekGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/merging_iter.go:842
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/merging_iter.go:911
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/iterator.go:1053
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/pebble_iterator.go:288
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/pebble_mvcc_scanner.go:407
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/mvcc.go:770
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/mvcc.go:730
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/storage/mvcc.go:618
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan.(*AbortSpan).Get
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/abortspan/abortspan.go:98
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.checkIfTxnAborted
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica.go:1966
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_evaluate.go:201
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_write.go:644
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_write.go:612
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_write.go:428
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_proposal.go:729
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_proposal.go:829
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_raft.go:113
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_write.go:167
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_send.go:460
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_send.go:156
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_send.go:197
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:191
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/server/node.go:1011
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +  |   /work/oxidecomputer/cockroach/pkg/server/node.go:994
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +Wraps: (4) pebble/table: invalid table 14889595 (checksum mismatch at 4168591/548677)
E251113 19:56:56.909466 27553747 1@util/log/logcrash/crash_reporting.go:174 ⋮ [-] 108191 +Error types: (1) *withstack.withStack (2) *markers.withMark (3) *withstack.withStack (4) *errutil.leafError

We have a core file available at: /staff/rack3/BRM42220077/2025-11-07/core.oxz_cockroachdb_371fba3a-658b-469b-b675-c90cc0d39254.cockroach.16312.1762260803 and I can transfer off further cores if needed.

Log files from the two new crashes detected today are in: /staff/rack3/BRM42220015/2025-11-20 and /staff/rack3/BRM42220083

[askfongjojo]

The cockroachdb peer on rack2 sled 14 panicked last night, also due to checksum mismatch. I've copied the core file and corresponding SMF log file to /staff/rack3/BRM42220077/2025-11-07/rack2-cockroach-panic.

* Wraps: (4) pebble/table: invalid table 14266170 (checksum mismatch at 6509871/417221)
* Error types: (1) *withstack.withStack (2) *markers.withMark (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: a SQL panic has occurred while executing the following statement:
* SELECT disk.id, disk.name, disk.description, disk.time_created, disk.time_modified, disk.time_deleted, disk.rcgen, disk.project_id, disk.disk_state, disk.attach_instance_id, disk.state_generation, disk.time_state_updated, disk.slot, disk.size_bytes, disk.block_size, disk.disk_type, virtual_provisioning_resource.id, virtual_provisioning_resource.time_modified, virtual_provisioning_resource.resource_type, virtual_provisioning_resource.virtual_disk_bytes_provisioned, virtual_provisioning_resource.cpus_provisioned, virtual_provisioning_resource.ram_provisioned, volume.id, volume.time_created, volume.time_modified, volume.time_deleted, volume.rcgen, volume.data, volume.resources_to_clean_up FROM (((disk INNER JOIN disk_type_crucible ON (disk_type_crucible.disk_id = disk.id)) LEFT JOIN virtual_provisioning_resource ON (virtual_provisioning_resource.id = disk.id)) LEFT JOIN volume ON (disk_type_crucible.volume_id = volume.id)) WHERE (disk.time_deleted IS NOT NULL)
*

[askfongjojo]

Two more occurrences of the panic on rack2 today (sled 14 and sled 17, running on 8ae817f/ab2bc89):

* Wraps: (4) pebble/table: invalid table 14558697 (checksum mismatch at 5299870/400709)
* Error types: (1) *withstack.withStack (2) *markers.withMark (3) *withstack.withStack (4) *errutil.leafError

* Wraps: (4) pebble/table: invalid table 14550981 (checksum mismatch at 5880189/525893)
* Error types: (1) *withstack.withStack (2) *markers.withMark (3) *withstack.withStack (4) *errutil.leafError

[iliana]

This problem first occurring prior to R17 seems to exonerate the major toolchain bump we landed in R17.

This issue seems to be cockroachdb/cockroach#57893, which the conversation there indicates is a problem below cockroach / pebble (either kernel or hardware).

All of the four stack traces reported above look similar to the user-reported stack trace in that issue.

Per that thread:

If node restarting seems to recover things, it really does sound like a hardware-level issue. The checksum check should either always fail for an SST file written already, or always pass, as those files are immutable once written.

In both of the recent cases, the files are missing -- we would expect to see e.g. /data/14558697.sst present. It's unclear what is removing them, or if they ever successfully finished writing in the first place. I'm trying to dig into that, because it sure would be nice to see what exactly is corrupt about these files. The discussion on the issue seems to indicate that is not entirely abnormal?

A possible thing to look at backporting to pebble is something like cockroachdb/pebble@a45eb82, which looks for singular bit flips in the block that failed the checksum and adds that to the error message.

[sudomateo]

Taking a look per @iliana's request.

This problem first occurring prior to R17 seems to exonerate the #8831 we landed in R17.

I agree, this panic appears unrelated to the major toolchain bump.

In both of the recent cases, the files are missing -- we would expect to see e.g. /data/14558697.sst present. It's unclear what is removing them, or if they ever successfully finished writing in the first place.

I read cockroachdb/cockroach#57893 and I can see why the maintainers are pointing to a hardware problem but I also don't understand why the files are missing. Admittedly I don't know exactly how CockroachDB is meant to write these files in the first place. Regardless, @augustuswm and @askfongjojo, did these panics occur on an idling system or were updates being performed at the time that would stop/start CockroachDB nodes?

A possible thing to look at backporting to pebble is something like cockroachdb/pebble@a45eb82, which looks for singular bit flips in the block that failed the checksum and adds that to the error message.

This would be good to have but also likely not to help us in this situation if the files are missing to begin with. I haven't poked at CockroachDB on dogfood or colo yet, but I would be curious to connect to the DB console (the web UI) to see what telemetry CockroachDB is giving us about what happened yesterday. I'll likely poke around at that.

[askfongjojo]

did these panics occur on an idling system or were updates being performed at the time that would stop/start CockroachDB nodes?

No, the core files were found during pre-update checks, i.e. before, not during software update. The system is never idle because of the many background tasks it's doing. Inventory collection, for one, do inserts/deletes periodically.

[sudomateo]

No, the core files were found during pre-update checks, i.e. before, not during software update. The system is never idle because of the many background tasks it's doing. Inventory collection, for one, do inserts/deletes periodically.

Perfect! That's what I figured but wanted to confirm. When I said "idle" I really meant under normal operation.

---

@jmcarp, @iliana, and myself hopped on a call to connect to the CockroachDB web console and see if anything stuck out in the metrics. Since the web console listens on 127.0.0.1:8080 in the CockroachDB zone there wasn't a simple way to create some port forward from our local machine -> castle -> switch zone -> host global zone -> cockroach zone.

We ended up modifying /opt/oxide/lib/svc/manifest/cockroachdb.sh in the CockroachDB node on sled 14 with the following diff.

diff --git a/opt/oxide/lib/svc/manifest/cockroachdb.sh b/opt/oxide/lib/svc/manifest/cockroachdb.sh
index 1d33ef9..dd66c67 100644
--- a/opt/oxide/lib/svc/manifest/cockroachdb.sh
+++ b/opt/oxide/lib/svc/manifest/cockroachdb.sh
@@ -25,7 +25,7 @@ fi
 args=(
   '--insecure'
   '--listen-addr' "$LISTEN_ADDR"
-  '--http-addr' '127.0.0.1:8080'
+  '--http-addr' '[fd00:1122:3344:10b::3]:8080'
   '--store' "$DATASTORE"
   '--join' "$JOIN_ADDRS"
 )

Note that the IP address used in the above diff was retrieved using ipadm in the CockroachDB zone. Then we restarted the cockroachdb service using svcadm restart cockroachdb. That allowed us to use SSH to create a port forward to 8080.

Note

We should create some mechanism within omicron to ease the process of connecting to CockroachDB web console.

Unfortunately we weren't able to find anything the stuck out from the metrics so we spent some time discussing patches we'd like to add to CockroachDB as part of our half measures towards resolving this issue.