feat: add a randomizer proxy to test (some part of) the testsuite#29
Draft
laurentsenta wants to merge 12 commits intomainfrom
Draft
feat: add a randomizer proxy to test (some part of) the testsuite#29laurentsenta wants to merge 12 commits intomainfrom
laurentsenta wants to merge 12 commits intomainfrom
Conversation
Contributor
Author
Contributor
|
I like it! That's really clever ❤️ |
Contributor
|
Maybe we could run tests against random gateway a couple of times. Or rerun the tests that passed. Then, if we found a test case that passed on all the attempts, we'd know something's wrong. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The worst thing that could happen to us is introducing a bug that disables one or more checks.
An example: in Kubo, due to a typo in the test, we miss a bug ipfs/kubo#9651
It's not crazy to imagine something similar could happen here.
The suite would be green, but the code wouldn't be tested.
We also need extra care when we port kubo tests: we can't do red/green testing.
So as a test writer, I have to introduce bugs (typos to values, wrong payloads, etc) to start from a red test and make sure it's working.
This PR proposes a "randomizer", which lives in front of a gateway and randomizes its outputs (headers and body).
If we run the test suite against our randomizer, we expect every test to fail most of the time.
If the entire test suite passes, there is a problem.
Next Steps