kvdb: concurrency issue tracker

[guggero]

After merging #5484 we'll need to make sure that any DB code is aware of the reset/retry mechanism.

This issue is meant to keep track of all the possible problems that should be addressed to handle DB errors and retries more gracefully.

• failIdx in batch/batch.go (etcd: enable full remote database support #5484 (comment))
• assert that the retry mechanism is working in all cases by artificially running the code in the callback twice (comment 1, kvdb: concurrency issue tracker #5557 (comment))
• assert that query functions with a callback can handle duplicates if the read transaction is retried (comment 2, kvdb: concurrency issue tracker #5557 (comment))

[joostjager]

Linking #5643 (comment)

"I had an idea that could help with that: modify the code to artificially always run the code inside each transaction twice. It is then as if everything needs a retry. Itests, bench, etc, should still work."

[joostjager]

The other issue that came up in #5643 is that callers of query functions with a callback such as ChannelGraph.ForEachChannel (and probably several others) should be prepared to receive duplicates if the read transaction is retried.

Linking #5484 (comment) about retries of a read-only tx.