reading from files being written

[joaomamede]

Do you think it's possible to read data from files from experiments currently being acquired(extremely long time lapses?).

The idea would be to manually shape the reader.

We want to analyze the data while it's being acquired to do conditional imaging.

Thanks.

[rbnvrw]

@joaomamede that is an interesting idea! Currently that is hard because of the way the ND2 file format is structured.

There is a header at the start, then all the data and at the end, there is a chunk of data known as the 'label_map' (see https://github.com/rbnvrw/nd2reader/blob/e44d392ba774dda1748c21ccdd518c5896907894/nd2reader/label_map.py) which contains info on where the data and metadata is stored in the file.

This LabelMap is currently used by the reader to retrieve the correct frame etc., but as it is written at the end of the file, I don't think it'll be present while the measurement is taking place.

See https://github.com/rbnvrw/nd2reader/blob/master/nd2reader/parser.py for details into how this works.

You could potentially guess the location of the data and then it would be possible, but I have never looked into this myself.
If you do investigate it further, please let me know what you find.

[joaomamede]

Thank you. It must be possible though as NIS elements allows for data reading while the experiment is going.

For either conditional imaging with HCA/jobs or for viewing the previous data while it's acquiring.

Any hint on your code (function where you read and seek) that I can mimic/explore?

I have little experience in binary file parsing.

[joaomamede]

I went digging into the parser, raw_metadata, Label.py etc.

I think the easiest way to do this would be one of these two options (so that we use the code that you wrote fully):

• Have a file of exactly the same size and structure and basically use it to grab the last portion of the file that gives the metadata ( the raw_text variable).
• have a pre assembled raw_text variable that is fed into the Parser class.
  Basically, using the metadata of an equal experiment but, using the data of the file being currently written.

To make it easier we could ask if a "filename_mock.nd2" exists, if it does get the metadata from such _fh file and then turn the reader back to _fh = open("filelname.nd2", "rb").
Either that or use a metadata.bin in the same folder for the same purpose (and we can make a call in Parser
def Parser(obect).....
def __init(self,fh, liveMeta)

What do you think, bad idea?

[joaomamede]

I forked nd2reader trying to get this working.

So far it, it is not working. The plan I tried to implement is to give two filenames to the reader. One picks the metadata and the other the data. I think nd2reader got lost in a seeking the right starting point.

https://github.com/joaomamede/nd2readerlive