Repository navigation
bpo-29505: Add fuzz tests for float(str), int(str), unicode(str) #2878
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
8316e8d
Add basic fuzz tests for a few common builtin functions.
ssbr f47f875
Remove fuzzing of hash() per comment by kcc / oss-fuzz-team.
ssbr d112252
Move LLVMFuzzerTestOneInput into cpython and tweak how test discovery…
ssbr cb9cdc0
Move the fuzzer to C++ so that it builds.
ssbr 7028614
Run the fuzz smoke tests on a little more input, just for kicks.
ssbr 2b34f07
Make the _fuzz module optional.
ssbr f77be65
Actually run the fuzz tests...
ssbr 778c827
Use unittest.main instead of test.support.
ssbr bc3d33f
Use C-style comments. (For porting to Python 2.7 and C89).
ssbr dee024f
Fix build break I accidentally introduced in f77be65.
ssbr 9a00b23
Add a little detail on what the heck this stuff even is, in readme.
ssbr d542130
s/fuzzer.c/fuzzer.cpp/
ssbr 9f16dd4
fuzzer.cpp -> fuzzer.c in faith that I can make it build.
ssbr aa6d784
Make _fuzz required, so that tests never fail on a successful build.
ssbr fa0af73
Fix the windows test failure by just not testing on windows. :)
ssbr 4af5c11
NEWS entry thingermajig.
ssbr fb017ed
Blacklist test_fuzz from coverage tests. It seems to fail with a Memo…
ssbr c4eda5d
Remove outdated comment.
ssbr 83967f9
Fix incorrect mod for base (should be % 37), and document why we're t…
ssbr 52dccc2
Use more idiomatic NULL checks rather than PyErr_Occurred().
ssbr 4327fd9
Attempt to explain a little more why these exist / what the relations…
ssbr 6da8e97
Renamed _fuzz to _xxtestfuzz and cleanup.
gpshead 43620ae
don't skip test_fuzz (besides, it was renamed)
gpshead File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Renamed _fuzz to _xxtestfuzz and cleanup.
Testing it resulted in a few cleanups such as PY_SSIZE_T_CLEAN being required to avoid from crashing in an opt build.
- Loading branch information
commit 6da8e979475e5e1c9dc2ca58263338d0b33f1854
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| import faulthandler | ||
| import test.support | ||
| import unittest | ||
|
|
||
| _xxtestfuzz = test.support.import_module('_xxtestfuzz') | ||
|
|
||
|
|
||
| class TestFuzzer(unittest.TestCase): | ||
| """To keep our https://github.com/google/oss-fuzz API working.""" | ||
|
|
||
| def test_sample_input_smoke_test(self): | ||
| """This is only a regression test: Check that it doesn't crash.""" | ||
| _xxtestfuzz.run(b"") | ||
| _xxtestfuzz.run(b"\0") | ||
| _xxtestfuzz.run(b"{") | ||
| _xxtestfuzz.run(b" ") | ||
| _xxtestfuzz.run(b"x") | ||
| _xxtestfuzz.run(b"1") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| faulthandler.enable() | ||
| unittest.main() |
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really need this module? It's only usage seems to be for the tests, and it looks like it has as much code as the fuzzers themselves!
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, but this would be like 2 lines in Cython. It's just the typical boilerplate of extension modules, nothing to write home about, right?
We don't need it. The thing we need is to make sure that the fuzz tests are maintained and don't break. That is, they need to be in CI for Python, not just run in the fuzz suite in oss-fuzz.
For most ways it could break, it would be enough just to build them, but not run them. I think it is a good idea to also run them. Although there's a cost: the macro silliness to make them runnable individually and collectively probably makes it easy to screw up, so I'm not happy about it. (Every fuzz test name is repeated like four times.) The strategy you mentioned above would make it less error prone, at the cost of losing the ability to call them from a unit test like this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To me what I see in this code is pretty standard CPython API calls. That raises the question of if there is a good reason to check these in and keep them working as part of CPython or if this code should be a stand alone module available on pypi that people could use to launch fuzz tests of any Python interpreter it is built with. The answer to that question likely comes from answering how oss-fuzz the fuzz testing service works. If it monitors our github repo and reruns fuzz tests as things change in cpython, including these here may make sense. I think a comment describing the situation and why this exists here with links to relevant oss-fuzz documentation (either in here or in the _fuzz/README.rst) would help that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added a little to README.rst, but to summarize and expand here:
You're correct that oss-fuzz regularly pulls from CPython and runs the fuzz tests locally.
oss-fuzz prefers to have the tests be a tested part of the upstream project so that they don't break. Certainly not all of these fuzz tests are using the public API (e.g._Py_HashBytes), and there will be more in the future. Those are more likely to they might break in the future -- having them here ensures that breaking them would break the tests, and they will always be good to run, hopefully.
We could keep the public-API-using fuzz tests separate in oss-fuzz, but they wouldn't really like that, and that actually splits up the tests in a confusing way -- some would be here, some would be in the oss-fuzz repo. I think they're better off all being here.