Add random jitter to retries to oddly distribute them #7601
Description
Describe the suggested improvement
Is your improvement related to a problem? Please describe.
Recently, we had a support case where retries resulted in MySQL deadlocks. Different endpoint instances are concurrently handling different messages, all configured to start a new saga. Because InnoDB handles index pages differently when the key is auto-increment, concurrent operations resulted in a deadlock. One could argue that retries are there also to solve this kind of issue. In this specific case, instances were retrying the deadlocked messages simultaneously, and the behavior repeated itself continuously.
Describe the suggested solution
Ideally, if each retry is delayed by its configured delay plus a small amount of randomized jitter, the problem has much less chance of surfacing.
Describe alternatives you've considered
Users can achieve this by providing a custom retry policy.
Additional Context
Link to private support case: https://particular-software--c.vf.force.com/apex/Case?num=00102599