-
Notifications
You must be signed in to change notification settings - Fork 189
feat: add VARBINARY type support #812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
I'm supportive of this. Is their clear behavior when doing binary operations around length? I'm assuming a collection of functions will be coming after this? Have any examples? |
|
Thanks, @jacques-n . I was looking into this in more detail and I'm not sure truncation behavior will be consistent across engines. For example, in SQL Server, when converting to |
vbarua
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The introduction of this type make sense to me. I'm happy to have this merged in by itself and then add functions that use them later.
One thing that we should do @jcamachor is define the https://substrait.io/extensions/#function-signature-compound-names for it.
|
I found an internal need for this change, so I decided to update it myself thinking: "this should be quick and easy to update and merge", and came out thinking that we should update the requirements for adding a new type to include at least:
1 ends up forcing the implementation of the type parser needed to handle said types in extensions files, without which these changes can't really be used. 2 forces the implementation of the function test handling machinery to make it easy for users to add tests for functions using this type. I'm still interested in pushing this work forward, but I want to add 1 and 2 properly to make sure it's usable which will take me more than just an hour in the afternoon like I was thinking originally. |
|
@vbarua Large ➕ on requiring all new simple extension related features to be reflected in at least one example in the core extension set. |
westonpace
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we missing protobuf messages for varbinary literals?
| } | ||
|
|
||
| message VarBinary { | ||
| int32 length = 1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now that I see this, are we okay with the int32 length? lol i'm fine with it... probably we need another type for LOB. Protobuf is using unsigned int32 for the binary payload.
Substrait currently supports FIXEDBINARY and BINARY types. However, it does not support VARBINARY (BINARY VARYING), which is part of the ANSI SQL and supported by many other systems (SQL Server family, MySQL, Oracle, etc.)
This PR adds the
VARBINARYtype and corresponding literal to the spec.