Partial read of E57 file

[Ephraim-Bryski]

I would like to do a partial read of the e57 point cloud data at a specific location, avoiding the costly operation of reading the entire file. This is a follow-up to the last comment of #74, where you mentioned this e57 library could be used instead, as the rust library it's wrapping has the functionality. I'm not familiar with the underlying rust e57 library, but I imagine the skip method of PointCloudReaderSimple might be useful. Is there potential for this, and could this be on the roadmap? Thanks!

Edit: Although this might not be great for this library, I was also wondering if the e57 could be parsed directly in python itself without using a wrapper (I tried this a bit: I was able to access the xml, but I was having trouble reading the raw data at the specified locations).

[dancergraham]

Hello, I'm not very active with developing this library right now but I would definitely be interested in pull requests / working together on adding this functionality and might get around to it myself at some point !

Thinking about how this would look - This could maybe be added to the standard point reader by adding a chunksize keyword argument similar to pandas.read_csv. I think ideally this would return an iterator over chunks of points.

[dancergraham]

I was also wondering if the e57 could be parsed directly in python itself without using a wrapper

I wonder whether numpy has methods for this... there is https://numpy.org/doc/stable/reference/generated/numpy.fromfile.html
I guess in theory you could cut out just the bit of the file you want and read that using numpy.fromfile if you can specify the correct data type to match the endianness, etc...

[Ephraim-Bryski]

I took a look at your library, since the wrapper code itself is written in rust (I think), and I don't know any rust, while maybe I could take a shot, sadly I'm probably not a great person for the task.

For implementing directly in python, I'm thinking you could directly use python's binary read, as I feel I would have more direct control over how to construct the numbers for different data types. (Maybe there's some advantages to numpy, like efficiency though?)

I used BunnyInt32.e57 from libe57 as a test. I first read it using the e57 library, the first coordinate is:

[-0.07063, 0.04015, 0.001226]

I then took a stab at reading the data myself. I was able to access the xml (I could share the code if you'd like), and it includes the data structure of a coordinate:

<cartesianX type="ScaledInteger" minimum="-2147483648" maximum="2147483647" scale="9.9999999999999995e-007"/>
<cartesianY type="ScaledInteger" minimum="-2147483648" maximum="2147483647" scale="9.9999999999999995e-007"/>
<cartesianZ type="ScaledInteger" minimum="-2147483648" maximum="2147483647" scale="9.9999999999999995e-007"/>
<cartesianInvalidState type="Integer" minimum="0" maximum="1"/>

I then tried reading this first coordinate:

import math

start_data_offset = 48  # the first 48 bytes are for the header

# information on the scaled integer format
# they were manually taken from the xml and of course should be retrieved dynamically
minimum_value = -2147483648
maximum_value = 2147483647
scale = 10 * 10**-7

n_bits_per_int = math.log2(maximum_value-minimum_value+1)
assert n_bits_per_int.is_integer(), "should never have a noninteger value"
assert n_bits_per_int == 32, "this file is supposed to be 32bit"

n_bytes_per_int = int(n_bits_per_int/8)
n_bytes_per_coordinate = n_bytes_per_int * 3    # this doesnt include the invalidstate, i'm not sure how many bits this would be (only 1?)

def get_integer_value(int_bytes: bytes):
    
    """converts the scaled integer bytes into a number"""
    
    # the header was little endian, so my guess is the data is too
    unscaled_value = int.from_bytes(int_bytes, "little")
    # from docs for scaledInteger:  The actual data value is computed by multiplying the stored integer by the scale and adding the offset. 
    scaled_int_offset = minimum_value * scale
    value = unscaled_value * scale + scaled_int_offset
    return value

with open("BunnyInt32.e57","rb") as f:

    f.seek(start_data_offset)

    # read the first coordinate values
    x_bytes = f.read(n_bytes_per_int)
    y_bytes = f.read(n_bytes_per_int)
    z_bytes = f.read(n_bytes_per_int)
    
    x = get_integer_value(x_bytes)
    y = get_integer_value(y_bytes)
    z = get_integer_value(z_bytes)

sadly, the values did not match at all. In binary, it's:

[b'\x01\x00\x00\x00', b'\x00\x00\x00\x00', b'\x90\xa8\x05\x00']

which I converted to:
[-2147.483647, -2147.483648, -2147.112816]

I'm a bit stuck here, I'm not sure if there's some wrinkle to the data format that I'm missing...

[dancergraham]

Do you maybe need to read the scale and offset from the node itself not from the file header ?

[dancergraham]

I think there is zero offset in this file - looking in cloud compare the "Shifted Box Centre" is equal to the "Global Box Centre" and the xml section doesn't describe an offset. That still doesn't give the right answers for me though ...
I guess you will also have the challenge of trying to read a single bit of data for the invalidstate where python only wants to read complete bytes

[Ephraim-Bryski]

I think there is zero offset in this file - looking in cloud compare the "Shifted Box Centre" is equal to the "Global Box Centre" and the xml section doesn't describe an offset.

Nice, I think you're right, it's storing signed integers with a scale but no offset.

I guess you will also have the challenge of trying to read a single bit of data for the invalidstate where python only wants to read complete bytes

Yup, I think it would doable with formatting as binary strings (simpler but I'm guessing super inefficient), or some bit manipulation on the individual bytes, but that would be tricky to get right (at least for me).

I'm still stumped on the values not matching the actual coordinates. Maybe there's a place to ask about this, or maybe we are better off with a wrapper around the Rust library like you're doing...