Open
Conversation
clessig
reviewed
Feb 14, 2026
| @@ -0,0 +1,343 @@ | |||
| # (C) Copyright 2024 WeatherGenerator contributors. | |||
Collaborator
There was a problem hiding this comment.
The data reader should go into the extra packages, i.e. https://github.com/ecmwf/WeatherGenerator/tree/develop/packages/readers_extra
| dataset = DataReaderAnemoi | ||
| case "fesom": | ||
| dataset = DataReaderFesom | ||
| case "msg_lst": |
Collaborator
There was a problem hiding this comment.
Remove when the reader is in https://github.com/ecmwf/WeatherGenerator/tree/develop/packages/readers_extra
| @@ -0,0 +1,36 @@ | |||
| SEVIRI_LST : | |||
Collaborator
There was a problem hiding this comment.
Please remove from the PR. We need a separate repo for the configs and are in the process of creating it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR supersedes #1803 was based on
main. This rebased version is now based on the latestdevelopas requested. Changes to 1803 include configuration files.Changes to 1803:
Note: The integration test
test_train_multi_streamon the model directory currently fails.The rest of the description follows 1803.
I implemented a custom datareader for the SEVIRI Level 2 land surface temperature product (LST). It works on on a gridded (lat lon) datastore of full Seviri images at hourly temporal resolution. The datastore is loaded with xarray during initialisation and lazily in the workers to avoid deadlocks caused by forked dataloaders.
The datareader extracts regions of interest from an image using indices that are stored in metadata files. It subsamples in time and space, e.g. here reduces hourly data to 6-hourly time steps. It loads normalisation statistics from external metadata files. It incorporates geoinfo (e.g. here DEM) from the datastore or returns empty geoinfo if not specified in the config.
This datareader is added to the MultiStreamSampler. A config used for finetuning for prognostic forecasting is added to configs/streams/.
I documented the datareader implementation with medium level of detail here: https://gitlab.jsc.fz-juelich.de/hedgedoc/s/LvbrmAWvf#
Issue Number
This is not linked to an issue.
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60