itest flake: `async_bidirectional_payments` flaked with timeout on `etcd` on Travis

[bhandras]

Background

The async_bidirectional_payments test flaked on Travis when running with etcd (dbbackend=etcd):

test_harness.go:90: Failed: (async bidirectional payments): exited with error: 
        *errors.errorString payment error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
        /home/travis/gopath/src/github.com/lightningnetwork/lnd/lntest/itest/lnd_payment_test.go:526 (0x15008c5)
        	testBidirectionalAsyncPayments: t.Fatalf("payment error: %v", err)
        /home/travis/gopath/src/github.com/lightningnetwork/lnd/lntest/itest/test_harness.go:114 (0x14b58ce)
        	(*harnessTest).RunTestCase: testCase.test(h.lndHarness, h)
        /home/travis/gopath/src/github.com/lightningnetwork/lnd/lntest/itest/lnd_test.go:245 (0x1552892)
        	TestLightningNetworkDaemon.func4: ht.RunTestCase(testCase)
        /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/testing/testing.go:1193 (0x523d0f)
        	tRunner: fn(t)
        /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/asm_amd64.s:1371 (0x472201)
        	goexit: BYTE	$0x90	// NOP

Your environment

• version of lnd: 75f5b40
• logs:
  64-async_bidirectional_payments-Bob-025eeb3c.log
  63-async_bidirectional_payments-Alice-026f8dd8.log

[bhandras]

Seems like the embedded etcd instance failed for some reason, log is full of these:

{"level":"warn","ts":"2021-09-08T03:32:56.254Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005f88c0/#initially=[http://127.0.0.1:6945]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}

[bhandras]

To aid debugging this issue if happens again I added a PR that redirects etcd logs to individual files (with elevated log level): #5702

[joostjager]

I've had this before with the benchmark: #5392 (comment)

But perhaps this isn't helpful because it is just the generic error that you always get.

It did happen on a non-embedded etcd instance and it was obviously also a load test.

[Roasbeef]

{"level":"warn","ts":"2021-09-08T03:32:56.254Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005f88c0/#initially=[http://127.0.0.1:6945]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}

Seems like a request timeout issue? So maybe we need to increase the context timeout for the client on our end?

In the past we've seen this in the wild with very slow networked disks (IOPs started to get throttled) attached to a live instance.

[yyforyongyu]

fixed by #6825