feature: `SST1RSoXSDB` support scan repeat loading without averaging

[cbishop4]

New functionality has been added to SST1 to allow for multiple exposures at each energy point during an RSoXS scan (i.e., instead of doing a 10-second exposure, you can do 5 2-second exposures.). Currently, PHS loads these scans by averaging them together, so you cannot access each individual pattern for statistics.

However, an issue has come into play once I started to do this (and should have opened an issue a couple months ago). To explain, we will adopt the working example of 5 2-second exposures. Once the code, as currently rewritten by me (but not publicly available in PHS) loads the 5 scans separately, those 5 scans do not add or average to the same I vs. q profile as the data initially loaded by the averaging method.

(Illustration of the mean(5 individual scans) =/= mean value from loading will be added soon)

[PriyankaKetkarBNL]

I agree. I am not aware of extensive testing to verify that the individual images taken at each energy can be considered statistically "the same" (e.g., no sample damage, beam motion). I think having loadRun output the individual images and then perhaps having a separate function to do the summation/average as a next step could be beneficial.

I can't guarantee that I can work on/contribute to this issue very soon, but I'm documenting some recent developments and rough thoughts here for anyone (including future me) who might be interested contributing. Some of these might lead to separate issues.

First of all, there was an update in Tiled on March 13, 2025 where we added a new datastream called ["primary"]["data"]["Wide Angle CCD Detector_tiff_time_stamp"]. This provides a timestamp for every individual image. Before this, there was only one timestamp per energy. Based on this, I think the dark subtraction function should be updated. I think using a try/except statement could work where we try using the "Wide Angle CCD Detector_tiff_time_stamp" to find the closest dark, otherwise use the ["primary"]["data"]["time"] (or perhaps use this to estimate/interpolate timestamps of the individual images?) to find the closest dark for scans taken prior to March 2025.

Note, the timestamps in the "Wide Angle CCD Detector_tiff_time_stamp" stream follow the EPICS convention of having a reference date of 1990-01-01. However, the timestamps written out into the ["primary"]["data"]["time"] datastream follow the Unix convention of having a reference date of 1970-01-01. Thus, the timestamps in the "Wide Angle CCD Detector_tiff_time_stamp" stream must be shifted. Here's an example of how I did that.

scan_id = 94012
scan_raw = catalog[int(scan_id)]
epics_reference_date = datetime(1990, 1, 1)
unix_reference_date = datetime(1970, 1, 1)
time_shift = (epics_reference_date - unix_reference_date).total_seconds()
timestamps_camera = time_shift + np.array(scan_raw["primary"]["data"]["Wide Angle CCD Detector_tiff_time_stamp"].read())
timestamps_tiled = np.array(scan_raw["primary"]["data"]["time"].read())
print(timestamps_camera)
print(timestamps_tiled)

Second of all, I have run a series of test measurements on SBA-15 samples at the C edge. I'm curious how the statistics compare to your dataset, @cbishop4. The scan IDs are 94007 (0.1 s exposure time, 50 repeats, timeout error when using PyHyper locally but this will be investigated during the current April 2025 shutdown), 94008 (1 s exposure time, 5 repeats), 94009 (2.5 s exposure time, 2 repeats), 94012 (5 s exposure time, 1 repeat). @cbishop4, I added you as an experimenter to proposal 315993 where these scans were taken, and anyone else interested in accessing these scans can message me to add them to the proposal. These could be good to test loadRun developments. Possibly in a separate issue, it would be good to check how comparable the sums of these images are, all of which have a total exposure of 5 s per energy.

[pbeaucage]

There are a lot of points to discuss here, but I want to drill in on one because it is a potential data reliability issue: (bolding mine)

First of all, there was an update in Tiled on March 13, 2025 where we added a new datastream called ["primary"]["data"]["Wide Angle CCD Detector_tiff_time_stamp"]. This provides a timestamp for every individual image. Before this, there was only one timestamp per energy. Based on this, I think the dark subtraction function should be updated. I think using a try/except statement could work where we try using the "Wide Angle CCD Detector_tiff_time_stamp" to find the closest dark, otherwise use the ["primary"]["data"]["time"] (or perhaps use this to estimate/interpolate timestamps of the individual images?) to find the closest dark for scans taken prior to March 2025.

The dark frame preprocessor stores the dark with the image in a separate stream. I do not know what was changed in the uncoordinated re-write of the beamline code, but I suspect because dark subtraction still works that the darks stream is still there and still meaningful. Timestamps aren't used in the dark matching routine. If that has changed, this is a five alarm fire that needs immediate investigation. I hope I misunderstood here, and if I did, please disregard. But if I didn't please open a new issue for said five alarm fire and we can start working on it...

[PriyankaKetkarBNL]

The dark frame preprocessor stores the dark with the image in a separate stream. I do not know what was changed in the uncoordinated re-write of the beamline code, but I suspect because dark subtraction still works that the darks stream is still there and still meaningful.

Nothing about the dark frame preprocessor has changed in the beamline code to my knowledge, and the structure of the dark and light image data streams appears unchanged in Tiled.

Timestamps aren't used in the dark matching routine.

My comment was based on my attempt (probably 1+ year ago at this point) to understand how loadRun worked. Line 704 had given me the impression that timestamps were used to match light images to the closest dark image. Am I missing something? My understanding is that nothing's "broken" in PyHyper; I just think the "Wide Angle CCD Detector_tiff_time_stamp" could be more accurate to use than "time".

[pbeaucage]

Ah! I think I understand, and I can breathe a little now.

So, time/image time is used in dark matching but only relatively. (i.e., find the most recent dark) Because the dark images are taken on the same camera/Bluesky process, absolutely correct timestamps without DAQ chain lag (or with less lag) don't buy anything. They would be helpful in matching to outside processes, and we probably should add the image timestamp to metadata which will neatly make it an option for dimensioning of the resulting array. But the case you're describing would require two images where the bluesky timestamps and the AreaDetector timestamps were so different they resulted in the order of images being inverted. That would be require multithreading kind of complexities in bluesky.

[PriyankaKetkarBNL]

absolutely correct timestamps without DAQ chain lag (or with less lag) don't buy anything

Agreed it makes no difference for standard energy scans with one image per energy. But I think it could matter in some cases for scans with 2+ (and especially a very large number of) images per energy. Just reiterating my original comment below and adding some examples:

there was an update in Tiled on March 13, 2025 where we added a new datastream called ["primary"]["data"]["Wide Angle CCD Detector_tiff_time_stamp"]. This provides a timestamp for every individual image. Before this, there was only one timestamp per energy.

Scan ID 90723 from our December 2024 beam time (prior to the addition of "Wide Angle CCD Detector_tiff_time_stamp") was run at 12 energies with 50 images per energy (dim_0 has a length of 50). However, there are only 12 timestamps in the "time" datastream, so I assume PyHyper might assign all 50 images at one energy the same timestamp and find a dark image accordingly. These images had 0.1 s exposure time, so it's probably a fine approximation. But I could imagine that for longer exposure times per image, it could be better to assign the earlier of the 50 images to the dark image taken before them and the later of the 50 images to the dark taken after them based on which dark is closer in time. Scan ID 94008 has the same issue with the "time" datastream, but it now also has the "Wide Angle CCD Detector_tiff_time_stamp" datastream where each raw image gets its own timestamp. Just a note, Scan ID 94008 has dim_3 containing the multiple images per energy instead of dim_0, but both scan IDs 90723 and 94008 seem to load fine using loadRun, so it is likely not an issue.

But the case you're describing would require two images where the bluesky timestamps and the AreaDetector timestamps were so different they resulted in the order of images being inverted. That would be require multithreading kind of complexities in bluesky.

I'm not immediately following why the order of images would need to be inverted nor am I familiar enough with multithreading, but I'll think about that some more.

[pbeaucage]

This would be easier as a conversation, but the way repeats is implemented, there is no possible opportunity for the darkframes preprocessor to run mid-series. So there could never be a dark interspersed in such a series, so a single timestamp at the beginning will use the most recent dark, which is correct. Images reaching forward for the next dark is highly problematic because we don't know if configuration changed between the image and the dark (whereas looking back, we do). As far as I know no instrument "reaches forward" for darks like this.

WRT second point, if the images are all taken in the same order, then the appropriate dark is always clear. You could only be incorrect if you had something like:

(Where I is real image and D is dark)

Real sequence:
D1 - I1 - I2 - I3 - D2 - I4 - I5 - ...
(Images 1, 2, 3 properly use dark 1, 4, 5 use dark 2)
Based on delayed timestamps
D1 - I1 - I2 - D2 - I3 - I4 - I5 - ...

in this case, image 3 is now using the improper dark #2 instead of dark 1.

As long as timestamps preserve order (which they do, because the instrument always goes trigger -> readout -> save (wait) -> trigger) we are fine. If the instrument did not wait for the image to be saved before sending the next trigger, because it was multithreaded, you could have things out of order.

Hope this clarifies. This should all be documented somewhere (instrument docs/wiki?), it's important especially that there can never be darks during a repeat scan.

Note, a repeat scan is different from a time series. A time series could have a dark mid condition, but will also have unique Bluesky time stamp for each condition, because Bluesky is ordering the exposure (and not the detector internally).

[PriyankaKetkarBNL]

Okay, I think that's starting to make more sense. From a quick glance at the DarkFramePreprocessor inputs, I'm not familiar with why the configuration of the prior dark is different or more reliable than the future dark, but I'll think about that some more. Currently, we don't have scans where we change exposure times within the same scan, but if we implement that later, that might be a case where a future dark with a different exposure time is inappropriate for correcting prior light images, and my earlier comment would not be relevant.

there is no possible opportunity for the darkframes preprocessor to run mid-series. So there could never be a dark interspersed in such a series

I had been wondering if it would be worthwhile working with DSSI to adjust the preprocessor and/or repeats implementation to enable taking darks interspersed within a series of images at a single energy? I had thought it could address some of our concerns with not enough darks taken with large numbers of images per energy, but do you think that could cause other problems?

Happy to discuss more in conversation and/or create separate issues for the above points. Just mainly noting these thoughts so I don't forget and/or if someone else may have similar confusions.

[pbeaucage]

First point: the dark needs to change if any motor on the beamline moves (due to ambient light changes), if the exposure time changes (more common), etc. Those changes basically "raise a hand" and tell it to take a dark next. Rather than checking every motor on the beamline, we simply trust this code, and use the last dark, because any frame taken after anything changes will by necessity have a new dark.

Second point: no, this would be a very bad idea. The entire point of repeats is to remove the absurd amount of Bluesky/DAQ chain overhead by setting the n_images parameter in areadetector, which saves about 2-5 s per image. Your proposed solution would ruin the entire way that repeats works/reason it is a useful feature at all. It might be interesting to understand that overhead via benchmarking; I have never been able to figure out a reason why, and it seems likely that some of the overhead could be reduced.

[PriyankaKetkarBNL]

Makes sense, and thanks for the helpful explanations. For point 1, I'd be interested in documenting your explanation; is there a preference for whether this belongs as a comment or in the ReadTheDocs? Also, do changes like this typically go straight to branch --> PR, or do you recommend starting an issue for ~1-line documentation?

[pbeaucage]

I'm not sure that the minutae of how the instrument takes darks are really important for a general analysis and reduction package. It would be great to have in the docs for the beamline code, in the beamline wiki, or perhaps (if you really want) in a SST-1 "developer level" documentation page in the docs. It's important to keep user and developer docs separate so people aren't overwhelmed.

I prefer issue -> PR so that people can see what is planned and have a chance to comment. The ideal PR is a one-line PR; short, sweet, separate issue PRs are easy to understand and easy to review.