953 stochastic noise #2275

maxwhitemet · 2026-01-08T12:09:22Z

Addresses #953.

This PR adds the plugin, CLI, and tests for stochastic noise generation.

The acceptance tests show the impact, with data available here.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

cpelley · 2026-01-12T08:44:28Z

@maxwhitemet, could you merge/rebase master into your branch. The failing "CI Tests / Test-Coverage (pull_request)" should hopefully be addressed by the now merged #2273

There does appear to be a new issue potentially with the improver documentation building now though... (different issue)

cpelley · 2026-01-13T11:41:26Z

Ignore the test coverage failure. Fixed in #2282

gavinevans

Thanks @maxwhitemet 👍

I've added some comments below.

gavinevans · 2026-01-15T16:32:58Z

envs/environment_b.yml

  - geopandas
  - lightgbm>=4.0.0
  - numba
+  - pysteps


Just to say, we shouldn't be including pysteps in this environment. If this has been done to avoid unit test failures, then we just need to handle the potential lack of availability of pysteps in the environment by skipping the unit tests.

gavinevans · 2026-01-15T16:33:30Z

envs/environment_b.yml

  - geopandas
  - lightgbm>=4.0.0
  - numba
+  - pysteps


Can you update the docstring at the top of the file to record that the stochastic noise CLI isn't supported by this environment?

Added. Also added a couple full stops.

gavinevans · 2026-01-15T16:36:17Z

improver/cli/stochastic_noise.py

+#
+# This file is part of 'IMPROVER' and is released under the BSD 3-Clause license.
+# See LICENSE in the root of the repository for full licensing details.
+"""CLI to add stochastic noise to a dependence template cube using SSFT."""


You need to spell out SSFT here, as it's not a sufficiently well known concept i.e. Short-Space Fourier Transform.

gavinevans · 2026-01-15T16:36:52Z

improver/cli/stochastic_noise.py

+    threshold_units: str = "mm/hr",
+):
+    """
+    Add stochastic noise to a dependence template cube object using SSFT.


Please spell out SSFT: Short-Space Fourier Transform.

gavinevans · 2026-01-15T16:43:44Z

envs/conda-forge.yml

  - scipy=1.15
  - sphinx<9.0.0  # https://github.com/tox-dev/sphinx-autodoc-typehints/issues/586
  - pandas
+  - pysteps


This should be removed, as optional dependencies, like pysteps are intended to be included in this environment.

gavinevans · 2026-01-16T18:16:05Z

improver/precipitation/stochastic_noise.py

+
+        num_workers = min(len(template.coord("realization").points), os.cpu_count() - 1)
+
+        results = compute(*delay_results, scheduler="threads", num_workers=num_workers)


Just to note, I don't think that we've used dask like this in the improver codebase before, although I think this usage makes sense. This might be something to discuss.

gavinevans · 2026-01-16T18:26:11Z

improver/precipitation/stochastic_noise.py

+        # Set noise to zero where dependence template data is below threshold
+        # to prevent spurious wet pixels
+        dry_mask = template.data < self._convert_threshold_units(template)
+        noise.data[dry_mask] = 0


I can see that this is different to the intended implementation with the intended implementation deliberately making the stochastic noise negative, so that it only has an impact for grid points where there is no precipitation in the templates. It would be good to discuss this.

I have made the change suggested, though made the scaling optional via a scale_subthreshold_noise parameter.

gavinevans · 2026-01-16T18:31:33Z

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

+            [[3.5848932, 4.995262], [2.2589254, 6.5118866]],
+            [[3.859587, 5.289296], [2.5182567, 6.830268]],


I think there's a slight misunderstanding here, as the intended outputs here should just be the inputs (as none of the grid points in the input is 0 mm/hr).

I have modified the plugin and tests following our conversations on my misunderstanding.

gavinevans · 2026-01-16T18:33:12Z

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

+    assert result.data.dtype == np.float32
+
+    # Verify noise was added (output differs from input)
+    assert not np.allclose(result.data, simple_cube.data)


Probably you should check what the numbers are here, rather than what they are not.

gavinevans · 2026-01-16T18:51:40Z

improver/precipitation/stochastic_noise.py

+
+        return output_cube
+
+    def process(self, dependence_template: Cube) -> Cube:


I think the acceptance tests have actually highlighted an issue where the units of the Cube being passed in, aren't being preserved, so although the metadata in the Cube says "m/s", the values are actually mm/hr. This should be corrected, so that the values put into the output cube are in the correct units.

I think this is now fixed. Thank you for pointing this out.

maxwhitemet

Thank you for the review @gavinevans. I have now made the requested changes.

maxwhitemet · 2026-01-20T09:36:22Z

envs/conda-forge.yml

  - scipy=1.15
  - sphinx<9.0.0  # https://github.com/tox-dev/sphinx-autodoc-typehints/issues/586
  - pandas
+  - pysteps


maxwhitemet · 2026-01-20T09:37:54Z

envs/environment_b.yml

  - geopandas
  - lightgbm>=4.0.0
  - numba
+  - pysteps


maxwhitemet · 2026-01-20T09:39:57Z

envs/environment_b.yml

  - geopandas
  - lightgbm>=4.0.0
  - numba
+  - pysteps


Added. Also added a couple full stops.

maxwhitemet · 2026-01-20T09:45:25Z

improver/cli/stochastic_noise.py

+#
+# This file is part of 'IMPROVER' and is released under the BSD 3-Clause license.
+# See LICENSE in the root of the repository for full licensing details.
+"""CLI to add stochastic noise to a dependence template cube using SSFT."""


maxwhitemet · 2026-01-20T09:46:09Z

improver/cli/stochastic_noise.py

+    threshold_units: str = "mm/hr",
+):
+    """
+    Add stochastic noise to a dependence template cube object using SSFT.


maxwhitemet · 2026-01-20T11:08:11Z

improver/precipitation/stochastic_noise.py

+                    delayed(self.do_fft)(realization_data.astype(np.float32))
+                )
+
+        num_workers = min(len(template.coord("realization").points), os.cpu_count() - 1)


maxwhitemet · 2026-01-20T13:37:06Z

improver/precipitation/stochastic_noise.py

+
+        return output_cube
+
+    def process(self, dependence_template: Cube) -> Cube:


I think this is now fixed. Thank you for pointing this out.

maxwhitemet · 2026-01-20T15:22:08Z

improver/precipitation/stochastic_noise.py

+        # Set noise to zero where dependence template data is below threshold
+        # to prevent spurious wet pixels
+        dry_mask = template.data < self._convert_threshold_units(template)
+        noise.data[dry_mask] = 0


I have made the change suggested, though made the scaling optional via a scale_subthreshold_noise parameter.

maxwhitemet · 2026-01-21T11:27:43Z

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

+            [[3.5848932, 4.995262], [2.2589254, 6.5118866]],
+            [[3.859587, 5.289296], [2.5182567, 6.830268]],


I have modified the plugin and tests following our conversations on my misunderstanding.

maxwhitemet · 2026-01-21T11:32:59Z

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

+    assert result.data.dtype == np.float32
+
+    # Verify noise was added (output differs from input)
+    assert not np.allclose(result.data, simple_cube.data)


maxwhitemet mentioned this pull request Jan 8, 2026

Add acceptance test data for stochastic noise metoppv/improver_test_data#119

Open

maxwhitemet added 6 commits January 12, 2026 09:25

Initial creation of plugin

43f4b5f

Remove redundant unit conversion

6af5792

Add stochastic noise finalisations, including tests

aba7a64

Fix reStructuredText issue with Sphinx

3130aec

Add pysteps to Sphinx's mock imports

403a074

Add pytest to environments

897e0cb

maxwhitemet force-pushed the 953_stochastic_noise branch from 84d3c1b to 897e0cb Compare January 12, 2026 09:56

gavinevans requested changes Jan 16, 2026

View reviewed changes

maxwhitemet added 2 commits January 21, 2026 15:06

Undo environemnt changes

08001d9

Implement review feedback

a2487f9

maxwhitemet commented Jan 21, 2026

View reviewed changes


		num_workers = min(len(template.coord("realization").points), os.cpu_count() - 1)

		results = compute(*delay_results, scheduler="threads", num_workers=num_workers)

		[[3.5848932, 4.995262], [2.2589254, 6.5118866]],
		[[3.859587, 5.289296], [2.5182567, 6.830268]],


		return output_cube

		def process(self, dependence_template: Cube) -> Cube:

953 stochastic noise #2275

Are you sure you want to change the base?

953 stochastic noise #2275

Uh oh!

Conversation

maxwhitemet commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 13, 2026

Uh oh!

gavinevans left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

maxwhitemet commented Jan 8, 2026 •

edited

Loading

cpelley commented Jan 12, 2026 •

edited

Loading