Updates to handle large Monte Carlo runs

[bolliger32]

Thus far, pyCIAM has only been run on a couple quantiles of sea level rise per segment. In theory, it can be run on large ensembles of monte carlo SLR projections, as needed for probabilistic estimates of global damages (as used for SC-GHG calculations).

The last time that pyCIAM was run in this context was prior to the full development of the package, and the package is not fully set up to handle this context. This PR contains a number of updates enabling the execution in this large ensemble context (as well as some others)

• handle any base years used for spatial distribution of capital (K) and pop in SLIIDERS
• make the surge lookup table creation more error-resilient
• fix some bugs arising when a segment variable other than "seg_adm" is used
• Fix bugs in the I/O functions for case in which on-the-fly quantiles of SLR monte carlo values are used
• allow for chunking along other dimensions than just segment (e.g. scenario, ssp, sample) when chunks are too big for dask workers
• allow for user to specify names of some dimensions of SLR and econ input data
• improve performance when creating dataset templates so that dask doesn't use a lot of memory creating a chunked array with many chunks
• xarray now allows region="auto" in .to_zarr() so we use that and remove "save_to_zarr_region" custom wrapper that is no longer needed
• prepare for a bunch of upcoming changes to default behavior in xr.concat by explicitly specifying kwargs
• use "as_completed" for tracking mapped workflows of tasks that save to zarr, rather than client.gather, so that pre-empted or dying workers don't necessitate re-running tasks
• add a few logs (print statements actually) to identify completed stages of execute_pyciam
• fix some bugs in use of scen_mc_filter to specify only running certain sceanrios and monte carlo samples
• remove an erroneously added return statement in execute_pyciam that was being used for debugging
• allow for a unique quantile to be specified only for creation of refA (by default use the median)
• reduce memory footprint of _load_lslr_for_ciam
• allow for vsl to be specified explicitly in econ inputs
• fix dtypes of object coords in template zarrs to avoid warnings with Zarr 3
• reduce memory footprint of surge lookup array creation
• add new numpy >= 2 dep to pyproject.toml
• update environment.yml by removing all pins except python version (previous pins included numpy v1)

[Copilot]

Pull request overview

This PR updates pyCIAM to handle large Monte Carlo ensembles of sea level rise projections, addressing several bugs and performance issues that arose when running in this probabilistic context.

Key changes:

• Enhanced flexibility for handling variable base years, dimension names, and subset selections across SLR and economic input data
• Improved error resilience and memory efficiency in surge lookup table creation
• Migration from custom save_to_zarr_region to xarray's native region="auto" parameter

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
pyCIAM/utils.py	Added support for variable base years in spatial data, made seg_var and mc_dim parameters configurable, added documentation for refA
pyCIAM/surge/lookup.py	Enhanced surge lookup creation with configurable dimension names, improved dtype handling for Zarr 3 compatibility, added workflow completion checks
pyCIAM/run.py	Major refactoring to support flexible chunking, improved workflow tracking with as_completed, enhanced parameter handling for large ensembles
pyCIAM/io.py	Refactored SLR data loading to handle quantile computation on-the-fly, improved memory efficiency, updated selector handling

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.