BIP 391: Binary Output Descriptors

[seedhammer]

A BIP 174 PSBT may contain an extended key for deriving input and output
addresses. This document proposes an additional field for PSBTs to represent
arbitrary BIP 380 output script descriptors.

To support transfer of output descriptors outside signing flows, the proposal
makes the unsigned transaction optional.

[achow101]

I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.

If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.

It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred. I just don't see how this proposal is at all useful.

Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.

[seedhammer]

I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.

If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.

FWIW, the first draft was indeed PSBT with a different header: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022184.html. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022186.html suggested we add descriptors to PSBT proper, which seemed like a good idea.

Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?

It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred.

How can the scriptPubKey infer a descriptor that contains, say, a key expression that ends in '/*'?

I just don't see how this proposal is at all useful.

The point of the BIP is to transfer descriptors between devices, in particular between a coordinator and (resource constrained) signing devices. It can be viewed as a codec for wallet policies.

Today, there exists a plethora of formats for exchanging descriptors, all with downsides:

• Raw xpubs are underspecified (missing script type)
• The BlueWallet textual format is inefficient (base58 xpubs) and doesn't support arbitrary descriptors.
• Raw BIP-380 descriptors are also inefficient and lack metadata.
• BCR-2023-010 are efficient but rely on the more complex (d)CBOR encoding.
• JSON encoded textual descriptors are also inefficient and require a JSON implementation.

This proposal addresses them all:

• Based on PSBT, signing devices need not spend additional code space on a separate codec.
• Efficient encoding of extended keys.
• Metadata (title, birth block) supported.
• Every current or future bip-380 descriptor can be represented.

Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.

We believe it's hopeless to keep a binary representation of output descriptors up to date with the moving target that is textual bip-380 descriptors (BCR-2020-010 attempted but failed). This BIP (along with wallet policies) address the worst offender by encoding extended keys efficiently.

[achow101]

Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?

That would be fine with me.

How can the scriptPubKey infer a descriptor that contains, say, a key expression that ends in '/*'?

Although /* cannot be directly inferred, you can still make an educated guess. It only affects the ends of derivation paths from xpubs, so the last derivation index of a key expression with an xpub could just be replaced with /*. Although descriptors allow for xpubs without a /*, it seems unlikely to me that that would actually be used in practice.

But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.

[pythcoiner]

But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.

By example in the current ledger miniscript signing flow, the hardware device produce a Proof of Registration (PoR) against the parent descriptor that is stored by the softwarewallet (in order to keep stateless) the first time the descriptor is checked by user on signer side, then every time the software wallet send a psbt to sign to ledger device, it should also supply also the parent descriptor and PoR.

we previously discuss about this here

[pythcoiner]

cc @bigspider

[bigspider]

Hardware signing devices need knowledge of the full descriptor/wallet policy in order to:

• recognize change addresses
• recognize from which of your different accounts (sometimes called wallets) the inputs are spending from.

(The second could perhaps be considered a strengthening of the traditional guarantees of hardware signers, not sure if it was ever formalized before wallet policies; I think it's essential in practice for anything that is not single-user)

While it's true that PSBTs today have enough info to sign, they do not have enough info to do it securely - the descriptor is also needed. Therefore, I agree that it would be a great improvement to be able to pass this info via PSBTs; currently, the Ledger app's signing flow is sign(PSBT, wallet_policy), and quite some work in the app goes into figuring out change/address_index for each input and change output based on the derivations in the PSBT.

However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.

[jonatack]

@seedhammer, do you plan to update here based on the review feedback?

[seedhammer]

Thanks for the prod, @jonatack . I've updated the proposal so it's no longer a PSBT extension but a separate format merely based on PSBT key-value maps.

I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at wizardsardine/liana#539 (comment), where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.

[pythcoiner]

I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at wizardsardine/liana#539 (comment), where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.

As i already said in previous discussion, i do not have strong opinion about wether the PoR should be standardized or not, but i do not think it should be standardized in this BIP.

And anyway if i get it well both sheme (HMAC vs Signature) will be 32 bytes of data?

[pythcoiner]

it should be interesting to have insight from other signing devices teams.

cc @benma @scgbckbone @stepansnigirev @stepansnigirev @JamieDriver

[seedhammer]

Also @kdmukai. Maybe others?

However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.

I want to point out that while saving space is important to us, the ability to add metadata and eventually proof-of-registration is the more interesting feature in general.

[benma]

I am in agreement with @bigspider's comment above.

BitBox02 libraries have functions like sign_psbt(psbt, wallet_policy) (example), and the policy/descriptor is required for anything but simple single-sigs.

It would be helpful if the way the policy is described and transferred to signers was standardized so apps don't have to implement different ways to interface with different signers.

PSBT seems like the obvious location to put this information. Another format that contains both the PSBT and the policy, which is interoperable with all software, seems difficult to achieve and impractical.

[seedhammer]

Given

• PSBTs are useful without descriptors, and
• descriptors are useful without PSBTs (backups, registration), and
• the combination of a PSBT and descriptor is useful for more secure signing flows,

Perhaps it's feasible to keep the formats separate, but concatenate them at the transport layer (QR code, files), in a backwards compatible way? Say, append the decriptor at the end of the PSBT, where devices without knowledge of descriptors may parse the PSBT and ignore the rest?

[murchandamus]

I have only lightly reviewed this document.

It sounds to me that there is still some discussion what solution is exactly to be pursued for this document. I’m going to change this to a draft PR for the moment, please set it ready to merge when the proposal is a bit more mature.

Please note that you will need a Backwards Compatibility section, and without going into the content in detail, it seems to me that the Rationale section could be a bit more extensive.

[seedhammer]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.

It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.

Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

[pythcoiner]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.

It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.

Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

To be more accurate, i think how is processed the PoR should de defined in a separate BIP, but having an optional key-value pair defined in this BIP should be useful imho.

[seedhammer]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.
It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.
Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

To be more accurate, i think how is processed the PoR should de defined in a separate BIP, but having an optional key-value pair defined in this BIP should be useful imho.

Good point. Done.

[scgbckbone]

I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.

Regardless of the format here (which i haven't studied) it is useful to have descriptor (or at least policy) in PSBT. Example from Coldcard is when you want to have multiple miniscripts registered on device that have identical keys in miniscript but policy/script is different. Or same keys same policy but order of keys in policy is different. Here you just cannot infer solely from PSBT which wallet to use because keys match for all wallets...

why not just use PSBT_GLOBAL_PROPRIETARY key and send descriptor in there?

[seedhammer]

I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.

Descriptors in PSBTs are useful, but descriptors are also useful outside PSBTs: transferring descriptors between wallets, (long term) backups. It would be unfortunate to specify a descriptor format that only solved the signing use-cases.

why not just use PSBT_GLOBAL_PROPRIETARY key and send descriptor in there?

That would hijack PSBT_GLOBAL_PROPRIETARY, leaving it no longer available for vendor-specific uses.

[jonatack]

A few review suggestions follow for the BIP text.

[murchandamus]

@seedhammer, any update on this proposal?

[seedhammer]

@jonatack yes, the title is good, thank you.

Assuming Craig is willing to implement the completed format in his coordinator software, I believe the next step is to find a hardware signer vendor to OK the proposal (in principle) and work with them to implement it.

We've been too busy to lead the process, and as such the BIP is in limbo. Can it be left in draft, or should it be closed?

[murchandamus]

@seedhammer: Revisiting this, it is not required for a BIP to be implemented before it can be merged in Draft status. If you feel that your proposal is at a point where you have no further planned work and the next step is adoption, it seems that this PR should be ready for Editor review.

[seedhammer]

SGTM. Thanks.

[murchandamus]

Whenever you are ready for editor review, please use the "Ready for review" button:

[pythcoiner]

I guess the block height is a fixed length here? Else how to know the length of the descriptor name?

[murchandamus]

Thank you for your patience. My apologies for losing track of this pull request being in our court. This proposal looks very mature. Let’s call it BIP 391. If there isn’t any more review or planned work in a few days, we can aim to publish it.

I had a similar misunderstanding as @benma when reading this, and was wondering if you might want to explicitly state either in the Abstract or at the start of the Specification section that this data structure is meant to store exactly one descriptor.

[murchandamus]

Could you please also add an entry for your proposal to the table in the README?

[seedhammer]

I've clarified that the format contains one descriptor, renamed the proposal to bip-0391.mediawiki, revised the preamble and added it to the README.

[craigraw]

This format has already stepped beyond what is strictly defined by an output descriptor through inclusion of a birthday block height. Given this you may wish to also consider other practical wallet recovery metadata such as

• Derivation path ranges (to avoid gap limit issues in recovering HD wallets)
• Silent payments labels (labels beyond 0)

That said, I actually think it would be better to extend the output descriptor language with optional key/value parameters appended to the end so we have an extensible format as new practical requirements arise. I've described such a scheme briefly in this comment: #2047 (comment). In this case, including the birthday as a separate field here would be redundant and/or confusing.

[murchandamus]

@seedhammer: I was planning on publishing this document if there wasn’t any more activity for a couple days, so please let me know whether you want to make changes before it is published as Draft due to Craig’s feedback or whether you prefer to proceed with publication. Either way, a BIP in Draft could still be adjusted after publication by another pull request.

[seedhammer]

Thank you @murchandamus . @craigraw points out that the question of where metadata belongs is not settled, so it seems to me best to put this proposal on hold. If consensus later decides that metadata should be kept entirely separate from output descriptors, we can revive this proposal.
Alternatively, I could remove metadata from the proposal, leaving just the encoding of the output descriptor and keys. @craigraw is a binary output descriptor encoding worth it at this point?

[craigraw]

is a binary output descriptor encoding worth it at this point?

Yes, I believe a standardised format is still useful.

However wrt to this proposal:

• I've proposed descriptor annotations for providing metadata to aid in recovering wallet funds. If adopted, this removes the need to for providing the block height as a separate field here, while providing other useful metadata such as a gap limit.
• I think consideration for other key expressions is needed. For example it is not possible to express a silent payments wallet with xpubs as the scan key must be private. See the proposal for silent payment output descriptors.