Some minor LND updates mostly etcd related

[bhandras]

Hi Bottlepay,

Ran a few benchmarks and for the etcd case it seems like a bunch CPU of time is spent in the TLS layer. I thought it'd be an interesting comparison vs the experimental postgres kvdb driver since it also doesn't use TLS at the moment. PTAL.

For reference, on my Hetzner cloud instance (AX-41-NVMe) the first few data points are:

lnd-etcd:
loadtest_1    | 2021-06-09T15:34:50.613Z	INFO	Speed	{"tps": 34.91778994137819, "count": 1000, "avg_latency_sec": 2.173053539867}
loadtest_1    | 2021-06-09T15:35:17.765Z	INFO	Speed	{"tps": 36.82955991150079, "count": 2000, "avg_latency_sec": 2.65595066485}
loadtest_1    | 2021-06-09T15:35:49.353Z	INFO	Speed	{"tps": 31.65781099648854, "count": 3000, "avg_latency_sec": 3.169215847652}
loadtest_1    | 2021-06-09T15:36:27.265Z	INFO	Speed	{"tps": 26.377218947832528, "count": 4000, "avg_latency_sec": 3.757643285047}

lnd-etcd-notls:
loadtest_1    | 2021-06-09T15:37:41.833Z	INFO	Speed	{"tps": 48.286619587525784, "count": 1000, "avg_latency_sec": 1.71128064486}
loadtest_1    | 2021-06-09T15:38:04.166Z	INFO	Speed	{"tps": 44.77649111135035, "count": 2000, "avg_latency_sec": 2.2084367948590002}
loadtest_1    | 2021-06-09T15:38:30.885Z	INFO	Speed	{"tps": 37.426390145914446, "count": 3000, "avg_latency_sec": 2.6139758464010003}
loadtest_1    | 2021-06-09T15:39:02.952Z	INFO	Speed	{"tps": 31.18514511811111, "count": 4000, "avg_latency_sec": 3.1428967239600003}

lnd-bbolt:
loadtest_1   | 2021-06-09T15:39:52.502Z	INFO	Speed	{"tps": 68.06800270016427, "count": 1000, "avg_latency_sec": 1.340028921759}
loadtest_1   | 2021-06-09T15:40:06.561Z	INFO	Speed	{"tps": 71.13055166118907, "count": 2000, "avg_latency_sec": 1.409935852396}
loadtest_1   | 2021-06-09T15:40:20.167Z	INFO	Speed	{"tps": 73.50000971254853, "count": 3000, "avg_latency_sec": 1.3745317160269999}
loadtest_1   | 2021-06-09T15:40:43.643Z	INFO	Speed	{"tps": 42.59649525501131, "count": 4000, "avg_latency_sec": 2.2255254259100004}

lnd-postgres:
loadtest_1   | 2021-06-09T15:52:31.981Z	INFO	Speed	{"tps": 21.06719191361456, "count": 100, "avg_latency_sec": 1.99804254599}
loadtest_1   | 2021-06-09T15:52:34.812Z	INFO	Speed	{"tps": 35.32848490623514, "count": 200, "avg_latency_sec": 3.55030612454}
loadtest_1   | 2021-06-09T15:52:37.536Z	INFO	Speed	{"tps": 36.70705569699947, "count": 300, "avg_latency_sec": 2.58857985994}
loadtest_1   | 2021-06-09T15:52:40.209Z	INFO	Speed	{"tps": 37.40881348735356, "count": 400, "avg_latency_sec": 2.9022524260300004}
loadtest_1   | 2021-06-09T15:52:43.210Z	INFO	Speed	{"tps": 33.32267165567664, "count": 500, "avg_latency_sec": 2.80426651104}
loadtest_1   | 2021-06-09T15:52:45.796Z	INFO	Speed	{"tps": 38.67153910837707, "count": 600, "avg_latency_sec": 2.92153166965}
loadtest_1   | 2021-06-09T15:52:48.616Z	INFO	Speed	{"tps": 35.46874487870118, "count": 700, "avg_latency_sec": 2.70489748709}
loadtest_1   | 2021-06-09T15:52:51.613Z	INFO	Speed	{"tps": 33.36672454905821, "count": 800, "avg_latency_sec": 2.72185303647}
loadtest_1   | 2021-06-09T15:52:54.555Z	INFO	Speed	{"tps": 33.98685100462618, "count": 900, "avg_latency_sec": 3.00046799454}
loadtest_1   | 2021-06-09T15:52:57.269Z	INFO	Speed	{"tps": 36.84741866480803, "count": 1000, "avg_latency_sec": 2.71027942725}
loadtest_1   | 2021-06-09T15:53:00.281Z	INFO	Speed	{"tps": 33.20699927487403, "count": 1100, "avg_latency_sec": 3.18886429944}
loadtest_1   | 2021-06-09T15:53:03.005Z	INFO	Speed	{"tps": 36.70633125047135, "count": 1200, "avg_latency_sec": 2.8672755618999997}
loadtest_1   | 2021-06-09T15:53:06.391Z	INFO	Speed	{"tps": 29.531323921056618, "count": 1300, "avg_latency_sec": 2.66793804251}
loadtest_1   | 2021-06-09T15:53:09.095Z	INFO	Speed	{"tps": 36.98161322747188, "count": 1400, "avg_latency_sec": 3.3551100755}
loadtest_1   | 2021-06-09T15:53:11.976Z	INFO	Speed	{"tps": 34.711252935983644, "count": 1500, "avg_latency_sec": 2.75368742209}
loadtest_1   | 2021-06-09T15:53:14.854Z	INFO	Speed	{"tps": 34.7556657771407, "count": 1600, "avg_latency_sec": 2.83816954941}
loadtest_1   | 2021-06-09T15:53:17.797Z	INFO	Speed	{"tps": 33.9809526939655, "count": 1700, "avg_latency_sec": 2.42738083576}
loadtest_1   | 2021-06-09T15:53:20.563Z	INFO	Speed	{"tps": 36.149760342199535, "count": 1800, "avg_latency_sec": 3.31638148345}
loadtest_1   | 2021-06-09T15:53:23.315Z	INFO	Speed	{"tps": 36.33376220264998, "count": 1900, "avg_latency_sec": 2.80917149742}
loadtest_1   | 2021-06-09T15:53:26.336Z	INFO	Speed	{"tps": 33.099810616719374, "count": 2000, "avg_latency_sec": 2.7922463510099997}
loadtest_1   | 2021-06-09T15:53:29.152Z	INFO	Speed	{"tps": 35.51604189112879, "count": 2100, "avg_latency_sec": 2.99062137092}
loadtest_1   | 2021-06-09T15:53:31.961Z	INFO	Speed	{"tps": 35.6000749270505, "count": 2200, "avg_latency_sec": 2.8415337365699997}
loadtest_1   | 2021-06-09T15:53:34.898Z	INFO	Speed	{"tps": 34.0499210727937, "count": 2300, "avg_latency_sec": 2.21473203568}
loadtest_1   | 2021-06-09T15:53:37.552Z	INFO	Speed	{"tps": 37.677034736194855, "count": 2400, "avg_latency_sec": 3.47598183528}
loadtest_1   | 2021-06-09T15:53:40.145Z	INFO	Speed	{"tps": 38.57444613242192, "count": 2500, "avg_latency_sec": 2.3575987787}
loadtest_1   | 2021-06-09T15:53:43.226Z	INFO	Speed	{"tps": 32.45418754796545, "count": 2600, "avg_latency_sec": 2.90371141602}
loadtest_1   | 2021-06-09T15:53:46.050Z	INFO	Speed	{"tps": 35.40567943091486, "count": 2700, "avg_latency_sec": 2.6694806509799998}

[joostjager]

Hi Andras, definitely a fair point about TLS. I ran your branch on our benchmarking rig (hardware described in the repo readme) and got the following result:

loadtest_1    | 2021-06-09T17:26:23.505Z        INFO    Speed   {"tps": 38.779088649031976, "count": 1000, "avg_latency_sec": 1.761182746682}
loadtest_1    | 2021-06-09T17:26:51.854Z        INFO    Speed   {"tps": 35.27464222534417, "count": 2000, "avg_latency_sec": 2.791405295284}
...
loadtest_1    | 2021-06-09T17:39:04.423Z        INFO    Speed   {"tps": 6.679227542767958, "count": 10000, "avg_latency_sec": 14.671612817518}
loadtest_1    | 2021-06-09T17:41:56.785Z        INFO    Speed   {"tps": 5.801735227733458, "count": 11000, "avg_latency_sec": 17.067011642481}
loadtest_1    | 2021-06-09T17:44:59.180Z        INFO    Speed   {"tps": 5.482626440647349, "count": 12000, "avg_latency_sec": 18.438811750786}
...
loadtest_1    | 2021-06-09T18:19:23.111Z        INFO    Speed   {"tps": 2.9829219133374973, "count": 20000, "avg_latency_sec": 32.508446747819}

The previous result of 4 tps was obtained after 10k payments. I see that it went up to 6 tps. One thing that I was wondering is whether there are also other changes in lnd 0.13 that increase etcd performance?

When the final release comes out, we can merge this PR and also update the remaining lnd configurations to 0.13.

Regarding etcd, it still seems to be the case that performance goes way down with larger number of payments, whereas with postgres tps remains stable.