Cache LLM requests for cheaper re-runs

[alistairjcbrown]

Let quick fixes land and be run more frequently by caching LLM responses. We'll need to make sure the cache key is related to the content of the prompt so that it gets a cache miss as expected whenever it changes