Skip to content

Conversation

@rbeucher
Copy link
Member

@rbeucher rbeucher commented Jan 8, 2026

This PR fixes how missing values are handled across the workflow, addressing the problems raised in issue #157.

The main idea is to treat missing data consistently from start to finish. Missing values are converted to NaN early on, so xarray can handle them naturally during calculations. Just before writing the final output, values are converted back to the CMIP6-expected missing value (for example 1e20). This avoids silent errors and inconsistent behaviour when processing data.

To support this, a small set of shared utilities was added to handle missing values in a consistent way, both in the core processing code and in the CMIP6 vocabulary handling. These utilities work with lazy datasets, so there is no unnecessary loading of large files.

As part of the cleanup, exception handling in one of the derivation utilities was also simplified by using standard Python errors instead of a custom exception.

@rbeucher rbeucher requested a review from rhaegar325 January 8, 2026 23:41
@codecov
Copy link

codecov bot commented Jan 8, 2026

Codecov Report

❌ Patch coverage is 60.97561% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.25%. Comparing base (bc36128) to head (f44eb7a).

Files with missing lines Patch % Lines
src/access_moppy/vocabulary_processors.py 60.82% 38 Missing ⚠️
src/access_moppy/base.py 76.19% 5 Missing ⚠️
src/access_moppy/derivations/calc_utils.py 0.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #162      +/-   ##
==========================================
+ Coverage   57.55%   58.25%   +0.70%     
==========================================
  Files          18       18              
  Lines        2403     2520     +117     
==========================================
+ Hits         1383     1468      +85     
- Misses       1020     1052      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants