XDCR Diff Tool Occasionally Stuck – Processing Rate Drops to Zero

[jayrossrccl]

We've observed an intermittent issue where the XDCR diff tool gets stuck during execution. The behavior is inconsistent—after re-running the diff, the issue typically resolves itself without any changes to the environment or script.

Symptoms:

• The target process appears to be stuck.
• Processing rate drops to 0 mutations/second, even though mutations are present.
• Example logs:

2025-08-08T08:02:08.749Z INFO GOXDCR.xdcrDiffTool: target processed 115812508 mutations, filtered 0 mutations, 0 failed filtering. processing rate=0 mutation/second
2025-08-08T08:02:13.749Z INFO GOXDCR.xdcrDiffTool: target processed 115812508 mutations, filtered 0 mutations, 0 failed filtering. processing rate=0 mutation/second
2025-08-08T08:02:18.749Z INFO GOXDCR.xdcrDiffTool: target processed 115812508 mutations, filtered 0 mutations, 0 failed filtering. processing rate=0 mutation/second

Questions:

• What might be causing the XDCR diff tool to get stuck?
• Why does the issue resolve itself after a re-run?
• How can we prevent this from happening in future executions?