[1710] power spectra

[enssow]

Description

Plots the power spectrum of the analysis increments
Adapted code for power spectra
for use with the WeatherGenerator model:

uv run packages/evaluate/src/weathergen/evaluate/example_extras/power_spectra/psd_main.py --run-id gn3gotvh --export-dir /p/home/jusers/owens1/juwels/WeatherGen/gn3gotvh --output-dir /p/home/jusers/owens1/juwels/WeatherGen/gn3gotvh/power_spectra

OR

uv run packages/evaluate/src/weathergen/evaluate/example_extras/power_spectra/psd_main.py --config packages/evaluate/src/weathergen/evaluate/example_extras/power_spectra/psd_config.yml

See design doc for more info: here

Issue Number

Closes #1710

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel

[enssow]

Prerequisties:

Please export the inference into a regular gridded netcdf first using the export package:
e.g.
uv run export --run-id <INFERENCE_ID> --stream ERA5 --output-dir ../output_nc --format netcdf --regrid-degree 1 --regrid-type regular_ll

[clessig]

Prerequisties:

Please export the inference into a regular gridded netcdf first using the export package: e.g. uv run export --run-id <INFERENCE_ID> --stream ERA5 --output-dir ../output_nc --format netcdf --regrid-degree 1 --regrid-type regular_ll

Does this also work if we regrid to regular 0.25 deg grid (e.g. when we work with the full resolution ERA5)?

[enssow]

Prerequisties:
Please export the inference into a regular gridded netcdf first using the export package: e.g. uv run export --run-id <INFERENCE_ID> --stream ERA5 --output-dir ../output_nc --format netcdf --regrid-degree 1 --regrid-type regular_ll

Does this also work if we regrid to regular 0.25 deg grid (e.g. when we work with the full resolution ERA5)?

Yes, it's possible to make it a 0.25 degree:

uv run export --run-id bhwuev7l --stream ERA5 --output-dir ../output_nc_bhwuev7l --format netcdf --regrid-degree 0.25 --regrid-type regular_ll --type prediction target
2026-02-06 10:48:00,930 - weathergen.evaluate.export.export_inference - INFO - {'run_id': 'bhwuev7l', 'type': ['prediction', 'target'], 'output_dir': '../output_nc_bhwuev7l', 'output_format': 'netcdf', 'stream': 'ERA5', 'fsteps': None, 'samples': None, 'channels': None, 'n_processes': 8, 'fstep_hours': 6, 'epoch': 0, 'rank': 0, 'quaver_template_folder': None, 'quaver_template_grid_type': 'O96', 'expver': None, 'regrid_degree': 0.25, 'regrid_type': 'regular_ll'}
2026-02-06 10:48:00,931 - weathergen.evaluate.export.export_inference - INFO - Starting processing prediction for run ID bhwuev7l. Detected None samples and None forecast steps.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:11<00:00, 35.74s/it]
2026-02-06 10:49:14,274 - weathergen.evaluate.export.export_inference - INFO - Finished processing prediction for run ID bhwuev7l.                                                              
2026-02-06 10:49:14,274 - weathergen.evaluate.export.export_inference - INFO - Starting processing target for run ID bhwuev7l. Detected None samples and None forecast steps.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:08<00:00, 34.26s/it]
2026-02-06 10:50:23,658 - weathergen.evaluate.export.export_inference - INFO - Finished processing target for run ID bhwuev7l.

It takes a bit longer to run the export than for 1 degree. But then you can generate the power spectra in the same way:

[enssow]

to investigate: default which relies on using _REPO_ROOT now doesn't point to the symlinked home repo but instead to the uv_cache_shared i.e. 'output_dir': PosixPath('/p/project1/weatherai/uv_cache_shared/environments-v2/psd-main-e0f8ffb71655b6d4/plots/power_spectra')

[anna-inf-met]

just placing this here - the difference in spectra with respect to regridding needs to be checked carefully as it is a known issue with reduced gaussian grid based models - not sure about healpix based ones! Keen to see the raw spectra - not ratio plots - for z,q,t,u,v at 850, 500 and 250 hPa.

[anna-inf-met]

to add - that 'bump' I suspect will be at the healpix artefact resolution size and directly related to it - for this specific model

[anna-inf-met]

Finally - there is some evidence emerging that fine tuning can remove these artefacts and 'bumps' in the spectra to a certain degree, for short lead times, for machine learning models generally. (Has been seen in healpix and regular grid based models)