Coalesce requests to prevent duplicate work #21
Description
A common caching feature a library like this should have is request coalescing, where multiple identical calls at the same time result in the cached work being done only once. At least conceptually it's as simple as putting a some kind of PENDING value into the cache in place of the result, and having any call that receives the PENDING value in place of a cached result, retry the cache with backoff. There may be edge cases that that particular solution doesn't handle well, but that's the idea.
Request coalescing is good for two kinds of situations. (1) When a ton of requests are made around the same time that require the same resources, request coalescing keeps the load of that off of the backing service when the cache is cold / expired. (2) Say a request takes 100 seconds to execute, if process A starts working on it at time 0 and process B asks for the result, with request coalescing it will take only 50 seconds for B (just waiting) instead of 100, and it'll also prevent the duplicate work.
Inspired by recommendations in https://aws.amazon.com/builders-library/caching-challenges-and-strategies