Add support for esplora backend#10538
Add support for esplora backend#10538niteshbalusu11 wants to merge 56 commits intolightningnetwork:masterfrom
Conversation
Refactor `handleNewHeader` to sequentially process blocks, ensuring complete block notification and watched address tracking. Key changes: - Process blocks from last known height to current - Fetch and validate each block header - Separate reorg detection from block processing - Improve logging and error handling - Ensure consistent block caching and notifications
This change adds REST API capabilities to the Electrum backend, enabling: - Fetching full block information - Retrieving transaction details - Finding transaction indices within blocks - Validating channel-related transactions Key changes include: - Added RESTClient to electrum package - Updated ElectrumNotifier and ChainClient to support REST URL - Modified configuration to require REST URL in Electrum mode - Implemented methods for block and transaction retrieval
This change adds support for fetching the actual pkScript for Taproot outputs when the script cannot be derived directly from the witness. It retrieves the full funding transaction to extract the correct output script for historical spend detection.
Ensure proper synchronization and order of operations when updating transaction notifier during block connected events. The height is now updated first, and confirmation/spend checks run concurrently to prevent potential race conditions.
This commit improves the fee estimation logic by: - Adding a method to find the most appropriate cached fee - Implementing fee clamping to ensure minimum fee thresholds - Handling edge cases like missing targets and exceeding max block target
Improve block transaction scanning by: - Increasing max concurrent block fetches - Using /block/:hash/txs endpoint for more efficient pre-fetching - Reducing API calls by processing addresses directly from API response - Minimizing duplicate transaction processing - Fetching only matched raw transactions at the end
The commit introduces an advanced gap limit scanning method for more efficient wallet recovery in the Esplora chain client. Key improvements include: - Configurable gap limit optimization - Ability to control address batch size - Dramatically faster wallet recovery by stopping scan after consecutive unused addresses - Parallel address querying for improved performance
Summary of ChangesHello @niteshbalusu11, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates a new Esplora backend for LND, providing a robust and efficient alternative to the existing Neutrino backend. The primary motivation is to enhance the user experience, particularly for users facing unreliable internet connections, by leveraging Esplora's HTTP-based API for blockchain interactions. The implementation covers essential services like block and transaction notifications, fee estimation, and wallet synchronization, with specific optimizations for faster and more reliable wallet recovery processes. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant new functionality by adding support for Esplora as a chain backend. This is a valuable addition, allowing lnd to operate in a lightweight mode without a local full node. The implementation is comprehensive, touching configuration, chain notification, chain view, and wallet logic, and includes thoughtful optimizations like gap-limit scanning for wallet recovery. My review focuses on potential bugs related to error handling, performance optimizations for sorting and data fetching, and adherence to the project's coding style. Overall, this is a great contribution.
| // createNewNotifier creates a new instance of the EsploraNotifier from a | ||
| // config. |
There was a problem hiding this comment.
According to the repository's style guide, function comments should start with the function's name. This comment should be updated to // CreateNewNotifier....
| // createNewNotifier creates a new instance of the EsploraNotifier from a | |
| // config. | |
| // CreateNewNotifier creates a new instance of the EsploraNotifier from a | |
| // config. |
References
- Function comments must begin with the function name. (link)
| func (e *EsploraFilteredChainView) filterBlockTransactions( | ||
| blockHeight uint32) []*wire.MsgTx { | ||
|
|
||
| e.filterMtx.RLock() | ||
| if len(e.chainFilter) == 0 { | ||
| e.filterMtx.RUnlock() | ||
| return nil | ||
| } | ||
|
|
||
| // Copy the current filter to avoid holding the lock during API calls. | ||
| watchedOutpoints := make(map[wire.OutPoint][]byte) | ||
| for op, script := range e.chainFilter { | ||
| watchedOutpoints[op] = script | ||
| } | ||
| e.filterMtx.RUnlock() | ||
|
|
||
| var filteredTxns []*wire.MsgTx | ||
| spentOutpoints := make([]wire.OutPoint, 0) | ||
|
|
||
| ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second) | ||
| defer cancel() | ||
|
|
||
| // For each watched outpoint, check if it was spent using the outspend | ||
| // endpoint. | ||
| for outpoint := range watchedOutpoints { | ||
| outSpend, err := e.client.GetTxOutSpend( | ||
| ctx, outpoint.Hash.String(), outpoint.Index, | ||
| ) | ||
| if err != nil { | ||
| log.Debugf("Failed to check outspend for %v: %v", | ||
| outpoint, err) | ||
| continue | ||
| } | ||
|
|
||
| if !outSpend.Spent { | ||
| continue | ||
| } | ||
|
|
||
| // Check if the spend is confirmed and at this block height. | ||
| if !outSpend.Status.Confirmed { | ||
| continue | ||
| } | ||
|
|
||
| if uint32(outSpend.Status.BlockHeight) != blockHeight { | ||
| continue | ||
| } | ||
|
|
||
| // Fetch the spending transaction. | ||
| tx, err := e.client.GetRawTransactionMsgTx(ctx, outSpend.TxID) | ||
| if err != nil { | ||
| log.Debugf("Failed to get spending tx %s: %v", | ||
| outSpend.TxID, err) | ||
| continue | ||
| } | ||
|
|
||
| filteredTxns = append(filteredTxns, tx) | ||
| spentOutpoints = append(spentOutpoints, outpoint) | ||
| } | ||
|
|
||
| // Remove spent outpoints from the filter. | ||
| if len(spentOutpoints) > 0 { | ||
| e.filterMtx.Lock() | ||
| for _, op := range spentOutpoints { | ||
| pkScript := e.chainFilter[op] | ||
| delete(e.chainFilter, op) | ||
|
|
||
| // Also remove from scripthash mapping. | ||
| if pkScript != nil { | ||
| sh := esplora.ScripthashFromScript(pkScript) | ||
| delete(e.scripthashToOutpoint, sh) | ||
| } | ||
| } | ||
| e.filterMtx.Unlock() | ||
| } | ||
|
|
||
| return filteredTxns | ||
| } |
There was a problem hiding this comment.
The current implementation of filterBlockTransactions iterates over all watched outpoints and makes a separate API call (GetTxOutSpend) for each one. For a node with many channels, this can result in a very large number of API calls per block, which is inefficient and may lead to being rate-limited by public Esplora servers.
A more efficient approach would be to fetch all transactions in the block with a single client.GetBlockTxs call (which handles pagination internally) and then iterate through the transactions and their inputs locally to check for spends of watched outpoints. This would significantly reduce the number of API requests.
There was a problem hiding this comment.
Made this more efficient.
|
Approved CI run |
Optimize scanning watched outpoints by fetching entire block transactions instead of making individual API calls.
|
@niteshbalusu11, remember to re-request review from reviewers when ready |
| if i < c.cfg.MaxRetries { | ||
| time.Sleep(time.Duration(i+1) * 100 * time.Millisecond) |
There was a problem hiding this comment.
it would be nice to turn the retry backoff from a linear to an exponential backoff, either configurable or by default. As is this is too aggressive for most backends.
Good to see that the max retry count is configurable though.
There was a problem hiding this comment.
also, I believe this call is only catching transport failures (no response at all). we should also try to catch non-200 errors eg. 429 (rate limiting) and 503 (service not available) in a block below this
| if i < c.cfg.MaxRetries { | ||
| time.Sleep(time.Duration(i+1) * 100 * time.Millisecond) |
There was a problem hiding this comment.
also, I believe this call is only catching transport failures (no response at all). we should also try to catch non-200 errors eg. 429 (rate limiting) and 503 (service not available) in a block below this
4 participants
Change Description
Description of change / link to associated issue.
Disclaimer: This PR was heavily written by AI but I spent a lot of time code reviewing and testing it
Motivation
I have been working on Blixt Wallet with @hsjoberg for a long time now and we have been using Neutrino as our backend, although neutrino is great for privacy, the UX of it with syncing issues has been a pain for a lot of our users especially for users in countries where internet connectivity is not very good. We even deployed btcd nodes in 4 continents to reduce latency for users but that hasn't helped either.
The PR introduces a new backend for LND based on esplora so you can connect to any public esplora endpoint such as mempool.space or blockstream.info. However, it is recommended that wallet devs run their own instance of esplora because the public endpoints are heavily rate limited.
Details
The core components of this PR are:
esplora/client.go: HTTP client for Esplora REST API with automatic retries and block polling for new blocks.esplora/chainclient.go: Implementschain.Interfacefor btcwallet integration, handling address watching, transaction notifications, and block processingesplora/fee_estimator.go: Implementschainfee.Estimatorinterface with cached fee estimates and automatic background updateschainntnfs/esploranotify/: Chain notifier implementation for confirmation and spend notificationsrouting/chainview/esplora.go: Filtered chain view for UTXO trackingWallet recovery:
Gap limitwhich I implemented which significantly improves the performance during recoveries.Block notification handling:
Configuration
Steps to Test
https://github.com/niteshbalusu11/lnd-esplora-testing
esploradirectory for important components.Also, @kaloudis helped a lot with testing this with Zeus on testnet.
Pull Request Checklist
Testing
Code Style and Documentation
[skip ci]in the commit message for small changes.📝 Please see our Contribution Guidelines for further guidance.