allow watchtower to store data in remote (etcd) backend

[calvinrzachman]

Hi, submitting this PR as an expression of interest in letting the watchtower server store data remotely. Verified that the tower writes to the remote storage using Polar and a local etcd cluster, but still need to look at what, if anything, needs adding from an automated testing perspective.

Never touched the DB before so feel free to let me have it on this one.

Could consider:

• putting the watchtower data in its own "namespace"/key prefix in etcd
• allowing separate configuration flag for remote watchtower storage

[guggero]

Thanks for the PR! Unfortunately your timing is a bit unlucky. We've actually started working on making all databases fully remote in etcd mode. So we're approaching this in a more generic manner.
I'll take a look at your code though, maybe we can re-use parts of it. I'm sorry if the PR itself might be superseded soon :/

[calvinrzachman]

Hi Oliver @guggero, that's great news! I probably should have asked in Slack before putting this one up. I was trying to take inspiration from what is done for the channeldb, but there is no doubt a way to consolidate some of the DB init/create logic. I am most interested to view the more generic PR once it is up. Thanks for the heads up.

[guggero]

Yeah, asking in an issue or on Slack generally helps to avoid this kind of situation. Really appreciate your effort though!

I just finished a first draft of the generic PR: #5484
Any kind of review/feedback is welcome. Just be aware that this is only part of a series of PRs to move forward with etcd, some performance optimizations around graph handling and path finding need to be implemented before this full remote mode can be used in practice.

[calvinrzachman]

Closing per guidance from Oliver above