bug/feature: Tiled-backed data loading performance and timeout issues

[pbeaucage]

SST1RSoXSDB has been timing out on loads of large scans through Tiled. This was first observed in December 2024 and discussed internally extensively with several work-arounds.

Summarizing briefly many discussions with NSLS2 DSSI/Tiled, the problem is that Dask's default configuration creates one thread per cpu core on the machine. Particularly on shared boxes with loads of cores, this easily swamps the load balancer and server cluster on the public-facing tiled.nsls2.bnl.gov. The fix outlined by Phil Maffetone is quite simple:

from dask.distributed import Client
from tiled.client import from_uri

c = from_uri("http://tiled.nsls2.bnl.gov")['csx']['raw']
run = c[202847]
da = run.primary['data']['axis1_image']

client = Client(
    n_workers=config["n_workers"],
    threads_per_worker=config["threads_per_worker"],
    processes=False,
)

arr = da.read() # Or da[:,:, :]
client.close()

# Do other work with arr

This should be integrated into SST1RSoXSDB, essentially creating a dask.distributed.Client in __init__ with some reasonable number of workers (perhaps the lesser of 12 threads or the number of cores). The client should be .close()'d on class destruction.

[pbeaucage]

Thread control turned out to not be the root cause of this problem, but rather illogical chunking of SST1 data where the default chunks are (10,10,790,790), (10,10,790,235), (10,10,235,790), (10,10,235,235). This causes each underlying tiff image to be read 4 times by 4 different servers leading to really astronomically slow loads.

The temporary workaround is to disable Tiled client-side timeouts, e.g. :

import PyHyperScattering, httpx
load = PyHyperScattering.load.SST1RSoXSDB(catalog_kwargs={'timeout'=httpx.Timeout(None)})

Long term, BNL will be pushing smarter chunking on their end to fix this. It may or may not be worth a PR to disable timeouts, or add a reraise exception if a timeout is thrown that explains how to disable timeouts.