Skip to content

Conversation

@syntrust
Copy link
Collaborator

@syntrust syntrust commented Jan 9, 2026

Addressing #483, this PR implemented more scanner modes for more choices. Check the doc for more details.

Tested hybrid scan modes on testnet with config:

--scanner.mode 7 \
--scanner.batch-size 1000 \
--scanner.interval.meta 10 \
--scanner.interval.blob 60 \
--scanner.interval.block 1440 \
INFO [01-06|12:08:59.534] Scanner update loop received event - all shards done.
INFO [01-06|12:08:59.534] Scanner started                          mode=check-meta+check-blob+check-block
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-meta                        interval=10m0s batchSize=1000
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-block                       interval=24h0m0s batchSize=7200
INFO [01-06|12:08:59.534] Local storage summary                    localKvs=shard0[0-17249] localKvCount=17250
INFO [01-06|12:08:59.534] Launching scan fix loop                  interval=12m0s
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-blob                        interval=1h0m0s  batchSize=1000
INFO [01-06|12:08:59.534] Scan batch started                       mode=check-meta                        startIndexOfKvIdx=0 kvsInBatch=[0-999]
INFO [01-06|12:08:59.560] Scan batch done                          mode=check-meta                        scanned=[0-999] count=1000 nextIndexOfKvIdx=1000 duration=26.427169ms
INFO [01-06|12:08:59.561] No last scanned block recorded, starting from 7200 slots ago
INFO [01-06|12:08:59.615] Latest updated KV indices fetched        startBlock=9,982,549 endBlock=9,989,749 totalUpdatedKvs=18 locallyStored=18
INFO [01-06|12:08:59.615] Scan batch started                       mode=check-block                       startIndexOfKvIdx=0 kvsInBatch=[17232-17249]
INFO [01-06|12:09:02.067] Scan batch done                          mode=check-block                       scanned=[17232-17249] count=18   nextIndexOfKvIdx=9,989,749 duration=2.507269601s
INFO [01-06|12:09:02.067] Scan batch started                       mode=check-blob                        startIndexOfKvIdx=0 kvsInBatch=[0-999]
INFO [01-06|12:09:49.486] Scan batch done                          mode=check-blob                        scanned=[0-999]       count=1000 nextIndexOfKvIdx=1000      duration=47.419013179s
INFO [01-06|12:09:59.535] Scanner stats                            mode=check-meta+check-blob+check-block mismatched=(none)
INFO [01-06|12:20:59.535] Scanner fix batch triggered
...
INFO [01-07|12:08:59.535] Local storage summary                    localKvs=shard0[0-17260] localKvCount=17261
INFO [01-07|12:08:59.535] Scanner stats                            mode=check-meta+check-blob+check-block mismatched=(none)
INFO [01-07|12:08:59.625] Latest updated KV indices fetched        startBlock=9,989,750 endBlock=9,996,058 totalUpdatedKvs=12 locallyStored=12
INFO [01-07|12:08:59.626] Scan batch started                       mode=check-block                       startIndexOfKvIdx=9,989,749 kvsInBatch=[17250-17254],[472],[17255-17260]
INFO [01-07|12:09:00.186] Scan batch done                          mode=check-block                       scanned=[17250-17254],[472],[17255-17260] count=12   nextIndexOfKvIdx=9,996,058 duration=652.352155ms
INFO [01-07|12:09:00.186] Scan batch started                       mode=check-blob                        startIndexOfKvIdx=5000      kvsInBatch=[5000-5999]
INFO [01-07|12:09:47.915] Scan batch done                          mode=check-blob                        scanned=[5000-5999]                       count=1000 nextIndexOfKvIdx=6000      duration=47.728831812s
INFO [01-07|12:09:47.915] Scan batch started                       mode=check-meta                        startIndexOfKvIdx=17000     kvsInBatch=[17000-17260]
INFO [01-07|12:09:47.927] Scan batch done                          mode=check-meta                        scanned=[17000-17260]                     count=261  nextIndexOfKvIdx=17261     duration=11.833072ms

Tested blob fix in local devnet:

INFO [01-13|18:20:58.393] Scan batch started                       mode=check-blob startIndexOfKvIdx=0 kvsInBatch=[160-207],[216-223]
WARN [01-13|18:21:03.647] Commit mismatch detected                 kvIndex=223 error="commit mismatch: expected=011ee3ee378e9349d1e239023f2c9bdb50b6e9648a78ea32, found=011d0a55a70937e244d598943e961ed2cf449d4d799ffb4d"
INFO [01-13|18:32:58.379] Scanner stats                            mode=check-blob mismatched=[223(pending)]
INFO [01-13|18:32:58.379] Scanner fix batch triggered
INFO [01-13|18:32:58.379] Scanner fixing batch                     mismatches=[223]
INFO [01-13|18:32:58.798] Fixing mismatched KV                     kvIndex=223
INFO [01-13|18:32:59.198] KV fixed successfully                    kvIndex=223
INFO [01-13|18:33:58.376] Scanner stats                            mode=check-blob mismatched=[]

@syntrust syntrust marked this pull request as ready for review January 13, 2026 10:57
@syntrust syntrust requested review from ping-ke, qizhou and qzhodl January 13, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants