Skip to content

Conversation

@jcamachor
Copy link
Contributor

Substrait currently supports FIXEDBINARY and BINARY types. However, it does not support VARBINARY (BINARY VARYING), which is part of the ANSI SQL and supported by many other systems (SQL Server family, MySQL, Oracle, etc.)

This PR adds the VARBINARY type and corresponding literal to the spec.

@jacques-n
Copy link
Contributor

I'm supportive of this. Is their clear behavior when doing binary operations around length? I'm assuming a collection of functions will be coming after this? Have any examples?

@jcamachor
Copy link
Contributor Author

Thanks, @jacques-n . I was looking into this in more detail and I'm not sure truncation behavior will be consistent across engines. For example, in SQL Server, when converting to VARBINARY from other types, padding/truncation happens on either the left or right depending on the source type (see here). So maybe we shouldn't define trimming behavior in the docs and instead leave it to the function implementations? But CAST is a special function type, so we probably need to define some standard behavior. As for useful functions, bitwise operations, encoding/decoding (e.g., base64), and cryptographic (MD5, SHA-2) could be good candidates.

Copy link
Member

@vbarua vbarua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The introduction of this type make sense to me. I'm happy to have this merged in by itself and then add functions that use them later.

One thing that we should do @jcamachor is define the https://substrait.io/extensions/#function-signature-compound-names for it.

@vbarua vbarua self-requested a review December 2, 2025 00:14
@vbarua
Copy link
Member

vbarua commented Dec 2, 2025

I found an internal need for this change, so I decided to update it myself thinking: "this should be quick and easy to update and merge", and came out thinking that we should update the requirements for adding a new type to include at least:

  1. A single function using the new types.
  2. Tests for any functions added.

1 ends up forcing the implementation of the type parser needed to handle said types in extensions files, without which these changes can't really be used. 2 forces the implementation of the function test handling machinery to make it easy for users to add tests for functions using this type.

I'm still interested in pushing this work forward, but I want to add 1 and 2 properly to make sure it's usable which will take me more than just an hour in the afternoon like I was thinking originally.

@benbellick
Copy link
Member

@vbarua Large ➕ on requiring all new simple extension related features to be reflected in at least one example in the core extension set.

Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we missing protobuf messages for varbinary literals?

}

message VarBinary {
int32 length = 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that I see this, are we okay with the int32 length? lol i'm fine with it... probably we need another type for LOB. Protobuf is using unsigned int32 for the binary payload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants