Allow users to override hardcoded `hypothesis` strategies #2066

pradeep90 · 2025-03-12T11:36:49Z

Override all the things!

The previous PR (#2061) allowed users to specify the container strategy. However, there's still a snag: the plugin hardcodes special strategies for TypeVar and Callable. These cannot be overridden by users, which is a problem when a law needs some generic value other than the container under test.

So, this PR allows users to override any of the strategies used by the plugin. It decouples the test-generation code from the hardcoded strategies by using a first-class value of Settings. The hardcoded strategies are in the default values of Settings and can be overridden without trouble.

Rationale

Why accept extra strategies within check_all_laws? Why not ask the user to register the strategies with hypothesis in the user's test file?

Our hypothesis plugin doesn't run tests. It creates tests and sticks them in the user's module.

So, even if I try:

with type_resolver.strategies_for_types({...}): 
     check_all_laws(MyContainer)

that sets the context only when defining the tests, not when running them. When the tests are run, the types are no longer mapped to the strategies. We could ask the user to set the context when pytest runs the tests, but that would be error-prone and tedious for the user. So, we have to set the context inside the created test. Hence this PR.

Why accept extra strategies?

Because the alternative is to write them to the hypothesis global variable, and global variables are evil.

Why can't the user register a strategy for KindN directly? It uses TypeVarTuple, which hypothesis doesn't understand. So, hypothesis thinks KindN is not a "subclass" of KindN[int, str, bool], because the number of type arguments don't match. Sad.

As a workaround, we need to write those kinds of laws as x: Callable[[], KindN[F, ...]] instead of x: KindN[F, ...]. That way, at least our Callable override can catch these types.

Implementation

laws.py:
- Replace the impure context managers with a pure Settings value.
- Split Settings into a default value (with the hardcoded strategies) and a user-provided value.
  - We merge these two values. (Monoid would have been useful for this - hint hint :) )
The rest is docs and tests.
- I've documented Settings inline. I believe it will be auto-included in the docs page.

Test Plan

Run test_custom_strategy_for_callable.py after adding print statements to see the generated examples. (I don't know of other ways to see the examples in this case, since there is so much dynamically added context.)

pytest returns/ tests/ -vv --log-level INFO -m "hypothesis" -rP --pdb -k "test_custom_strategy_for_callable"

Before

The parameter container: KindN[_F_Applicative, ...] is always bound to a _Wrapper even though _Wrapper is not even an Applicative! (This is because _Wrapper <: KindN, sadly.)

some_identity <Wrapper: None> container <Wrapper: None>
some_identity <Wrapper: 0> container <Wrapper: 0>
some_identity <Wrapper: -27749> container <Wrapper: -116>
some_identity <Wrapper: False> container <Wrapper: False>
some_identity <Wrapper: False> container <Wrapper: True>
some_identity <Wrapper: b''> container <Wrapper: b''>

After

Pass in a strategy for Callable. Now, we get container as Maybes and Results.

some_identity <Wrapper: 0> container <Nothing>
some_identity <Wrapper: 25634> container <Some: None>
some_identity <Wrapper: -16697> container <Success: b'v\xf0'>
some_identity <Wrapper: -16697> container <Success: None>
some_identity <Wrapper: -95> container <Some: 3>
some_identity <Wrapper: -95> container <Some: 0>
some_identity <Wrapper: 2> container <Success: 2>
some_identity <Wrapper: 2> container <Some: 2>
some_identity <Wrapper: -15533> container <Some: 1.5>
some_identity <Wrapper: -15533> container <Some: 1.5>
some_identity <Wrapper: 3> container <Success: 0.0>
some_identity <Wrapper: 3> container <Some: 0.0>
some_identity <Wrapper: 5739> container <Failure: 12655>

Checklist

I have double checked that there are no unrelated changes in this pull request (old patches, accidental config files, etc)
I have created at least one test case for the changes I have made
I have updated the documentation for the changes I have made
I have added my changes to the CHANGELOG.md

Did not know about this earlier. Move in the tests to `test_laws_resolution.py`, since they are testing `laws`, not anything about `hypothesis`.

Want to make this pure and user-overridable.

This package seems to put callers above callees.

Phew. Finally.

This way, the deeper code doesn't need to know about defaults and overrides. It just deals with `Settings`.

codecov · 2025-03-12T11:48:36Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (82ef3ef) to head (c1b78f2).
Report is 347 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##            master     #2066    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           80        81     +1     
  Lines         2485      2576    +91     
  Branches       437        44   -393     
==========================================
+ Hits          2485      2576    +91

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sobolevn

Thanks, the work you are doing is really impressive. Please, let me know when to make a new release.

sobolevn · 2025-03-12T13:34:41Z

docs/pages/contrib/hypothesis_plugins.rst

+  Checking laws is not compatible with ``pytest-xdist``,
+  because we use a lot of global mutable state there.
+  Please, use ``returns_lawful`` marker
+  to exclude them from ``pytest-xdist`` execution plan.


We can also make a note about running them on a single node.

Ah, this was an existing paragraph that I moved up. I don't have anything to add here.

sobolevn · 2025-03-12T13:37:39Z

tests/test_contrib/test_hypothesis/test_laws/test_custom_strategy_for_callable.py

+check_all_laws(
+    _Wrapper,
+    container_strategy=st.builds(_Wrapper, st.integers()),
+    other_strategies=other_strategies,


maybe register_strategies? other feels a bit off to me.

Agreed. How about type_strategies?

pradeep90 · 2025-03-12T14:06:46Z

Thanks, the work you are doing is really impressive. Please, let me know when to make a new release.

Thanks! I appreciate your speedy responses too. Really helps keep the cycle time low.

I'll test-drive it for another week or so, just to make sure I don't run into any more issues that need fixing :)

sobolevn

Thank you!

docs/pages/contrib/hypothesis_plugins.rst

pradeep90 · 2025-03-20T12:12:54Z

Please, let me know when to make a new release.

@sobolevn I didn't run into any more problems :) We should be good for a new release. Thanks!

sobolevn · 2025-03-20T19:01:11Z

Awesome! Scheduled for tomorrow.

sobolevn · 2025-03-21T16:16:11Z

0.25.0 is released 🎉
Thanks a lot for your contributions!

pradeep90 added 18 commits March 11, 2025 14:23

hypothesis: get lawful ancestors using the laws() classmethod.

2be1352

Did not know about this earlier. Move in the tests to `test_laws_resolution.py`, since they are testing `laws`, not anything about `hypothesis`.

hypothesis: extract function for registering strategies.

16bc815

Want to make this pure and user-overridable.

hypothesis: replace hypothesis mutation with context manager.

02a4f50

hypothesis: extract functions for strategy factories.

9f5ee66

hypothesis: inline context managers for pure functions and type vars.

448ae84

hypothesis: add test for hypothesis state with settings.

96106b1

hypothesis: inline context managers for container and interfaces.

f476ef8

hypothesis: merge the context mappings.

2957d5e

hypothesis: rearrange functions in calling order.

e093247

This package seems to put callers above callees.

hypothesis: extract a pure function for types-to-strategies.

9e7d609

Phew. Finally.

hypothesis: split settings into default and override.

cfc58fc

hypothesis: accept other strategies in check_all_laws.

bdccc04

hypothesis: add end-to-end test for registering callable.

d57d6b0

hypothesis: merge settings in check_all_laws.

db141ba

This way, the deeper code doesn't need to know about defaults and overrides. It just deals with `Settings`.

hypothesis: make Settings public and document it inline.

31669c2

hypothesis: add type test.

f465cc8

hypothesis: add docs.

17c39ee

hypothesis: fix typo.

ab6a362

sobolevn reviewed Mar 12, 2025

View reviewed changes

hypothesis: rename attribute and parameter for type strategies.

40da087

sobolevn approved these changes Mar 12, 2025

View reviewed changes

docs/pages/contrib/hypothesis_plugins.rst Show resolved Hide resolved

Update docs/pages/contrib/hypothesis_plugins.rst

c1b78f2

sobolevn merged commit 2f4f6b5 into dry-python:master Mar 12, 2025
23 of 24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow users to override hardcoded `hypothesis` strategies #2066

Allow users to override hardcoded `hypothesis` strategies #2066

Uh oh!

pradeep90 commented Mar 12, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 12, 2025 •

edited

Loading

Uh oh!

sobolevn left a comment

Uh oh!

sobolevn Mar 12, 2025

Uh oh!

pradeep90 Mar 12, 2025

Uh oh!

sobolevn Mar 12, 2025

Uh oh!

pradeep90 Mar 12, 2025

Uh oh!

pradeep90 commented Mar 12, 2025

Uh oh!

sobolevn left a comment

Uh oh!

Uh oh!

Uh oh!

pradeep90 commented Mar 20, 2025

Uh oh!

sobolevn commented Mar 20, 2025

Uh oh!

sobolevn commented Mar 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Allow users to override hardcoded hypothesis strategies #2066

Allow users to override hardcoded hypothesis strategies #2066

Uh oh!

Conversation

pradeep90 commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Override all the things!

Rationale

Why accept extra strategies?

Implementation

Test Plan

Before

After

Checklist

Uh oh!

codecov bot commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

sobolevn Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

pradeep90 Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

sobolevn Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

pradeep90 Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

pradeep90 commented Mar 12, 2025

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pradeep90 commented Mar 20, 2025

Uh oh!

sobolevn commented Mar 20, 2025

Uh oh!

sobolevn commented Mar 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Allow users to override hardcoded `hypothesis` strategies #2066

Allow users to override hardcoded `hypothesis` strategies #2066

pradeep90 commented Mar 12, 2025 •

edited

Loading

codecov bot commented Mar 12, 2025 •

edited

Loading