Occasional Exception in attributes.py to_byte_array()

[srl17]

I am currently using PyCryptoki 2.5.23

I have an IV that the HSM randomly generates. I call other functions such as a mechanism. It eventually funnels down to to_byte_array().
Works great most of the time. It seems about 1 out 500,000 calls to the HSM I get some random byte string such as b'UvG;H&,TGi*2' which doesn't meet any of checks for a byte string. It thus tries the value as a Hex String val = int(val, 16) which it isn't and then a ValueError exception is thrown. On the current master branch this is occurring on line 305.

I added in my usage a check if all the values are Hex and if they are then I say val = int(val, 16) otherwise I assume it is a standard bytestring like in this check 'if "\x" in repr(val):'

Just wanted to let you know about this low rate exception issue I have seen.

[astraw38]

Hey @srl17 , thanks for the report! That function is an attempt to keep backwards compatibility with some python2.7 tooling. Python2 had bytestring as the default for strings, which would lead to people passing in valid bytestrings that could be directly converted to a bytearray... yet it would be text data. This could lead to confusion when matched with other tooling that converted strings -> byte array first. Notably, this would happen for hex bytestrings, like deadbeef, which is both valid hex and valid bytestring, and a decision really can't be made between the two... so we default to assuming hex, which is why you are getting that error.

It's definitely an area of improvement. I'd like to drop python2 support altogether, which would greatly simplify that function into bytestring -> bytes; unicode -> (assume hex) -> bytes. I'd look to that in a major version bump though, since it might break some things.

As for a temporary solution for your usecase, I highly recommend using a bytearray, or a list of bytes. That will provide a consistent, explicit conversion, and ensures that users will put the data in a proper format first (no hex disguised as bytestring). If you're extracting the HSM-created IV from a c_mech, then do the additional step of converting to a list of bytes, rather than round-tripping bytestring. That's a simple list(v), presuming again that you're working with bytestring.