Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8316e8d
Add basic fuzz tests for a few common builtin functions.
ssbr Jul 21, 2017
f47f875
Remove fuzzing of hash() per comment by kcc / oss-fuzz-team.
ssbr Jul 25, 2017
d112252
Move LLVMFuzzerTestOneInput into cpython and tweak how test discovery…
ssbr Jul 25, 2017
cb9cdc0
Move the fuzzer to C++ so that it builds.
ssbr Jul 25, 2017
7028614
Run the fuzz smoke tests on a little more input, just for kicks.
ssbr Jul 26, 2017
2b34f07
Make the _fuzz module optional.
ssbr Jul 26, 2017
f77be65
Actually run the fuzz tests...
ssbr Jul 26, 2017
778c827
Use unittest.main instead of test.support.
ssbr Jul 26, 2017
bc3d33f
Use C-style comments. (For porting to Python 2.7 and C89).
ssbr Jul 27, 2017
dee024f
Fix build break I accidentally introduced in f77be65.
ssbr Jul 27, 2017
9a00b23
Add a little detail on what the heck this stuff even is, in readme.
ssbr Jul 27, 2017
d542130
s/fuzzer.c/fuzzer.cpp/
ssbr Jul 27, 2017
9f16dd4
fuzzer.cpp -> fuzzer.c in faith that I can make it build.
ssbr Aug 3, 2017
aa6d784
Make _fuzz required, so that tests never fail on a successful build.
ssbr Aug 23, 2017
fa0af73
Fix the windows test failure by just not testing on windows. :)
ssbr Aug 23, 2017
4af5c11
NEWS entry thingermajig.
ssbr Aug 24, 2017
fb017ed
Blacklist test_fuzz from coverage tests. It seems to fail with a Memo…
ssbr Aug 24, 2017
c4eda5d
Remove outdated comment.
ssbr Aug 24, 2017
83967f9
Fix incorrect mod for base (should be % 37), and document why we're t…
ssbr Aug 25, 2017
52dccc2
Use more idiomatic NULL checks rather than PyErr_Occurred().
ssbr Aug 25, 2017
4327fd9
Attempt to explain a little more why these exist / what the relations…
ssbr Aug 25, 2017
6da8e97
Renamed _fuzz to _xxtestfuzz and cleanup.
gpshead Sep 6, 2017
43620ae
don't skip test_fuzz (besides, it was renamed)
gpshead Sep 6, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Renamed _fuzz to _xxtestfuzz and cleanup.
Testing it resulted in a few cleanups such as PY_SSIZE_T_CLEAN being
required to avoid from crashing in an opt build.
  • Loading branch information
gpshead committed Sep 6, 2017
commit 6da8e979475e5e1c9dc2ca58263338d0b33f1854
21 changes: 0 additions & 21 deletions Lib/test/test_fuzz.py

This file was deleted.

23 changes: 23 additions & 0 deletions Lib/test/test_xxtestfuzz.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import faulthandler
import test.support
import unittest

_xxtestfuzz = test.support.import_module('_xxtestfuzz')


class TestFuzzer(unittest.TestCase):
"""To keep our https://github.com/google/oss-fuzz API working."""

def test_sample_input_smoke_test(self):
"""This is only a regression test: Check that it doesn't crash."""
_xxtestfuzz.run(b"")
_xxtestfuzz.run(b"\0")
_xxtestfuzz.run(b"{")
_xxtestfuzz.run(b" ")
_xxtestfuzz.run(b"x")
_xxtestfuzz.run(b"1")


if __name__ == "__main__":
faulthandler.enable()
unittest.main()
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#define PY_SSIZE_T_CLEAN
#include <Python.h>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this module? It's only usage seems to be for the tests, and it looks like it has as much code as the fuzzers themselves!

@ssbr ssbr Jul 26, 2017 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but this would be like 2 lines in Cython. It's just the typical boilerplate of extension modules, nothing to write home about, right?

We don't need it. The thing we need is to make sure that the fuzz tests are maintained and don't break. That is, they need to be in CI for Python, not just run in the fuzz suite in oss-fuzz.

For most ways it could break, it would be enough just to build them, but not run them. I think it is a good idea to also run them. Although there's a cost: the macro silliness to make them runnable individually and collectively probably makes it easy to screw up, so I'm not happy about it. (Every fuzz test name is repeated like four times.) The strategy you mentioned above would make it less error prone, at the cost of losing the ability to call them from a unit test like this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me what I see in this code is pretty standard CPython API calls. That raises the question of if there is a good reason to check these in and keep them working as part of CPython or if this code should be a stand alone module available on pypi that people could use to launch fuzz tests of any Python interpreter it is built with. The answer to that question likely comes from answering how oss-fuzz the fuzz testing service works. If it monitors our github repo and reruns fuzz tests as things change in cpython, including these here may make sense. I think a comment describing the situation and why this exists here with links to relevant oss-fuzz documentation (either in here or in the _fuzz/README.rst) would help that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a little to README.rst, but to summarize and expand here:

You're correct that oss-fuzz regularly pulls from CPython and runs the fuzz tests locally.

oss-fuzz prefers to have the tests be a tested part of the upstream project so that they don't break. Certainly not all of these fuzz tests are using the public API (e.g._Py_HashBytes), and there will be more in the future. Those are more likely to they might break in the future -- having them here ensures that breaking them would break the tests, and they will always be good to run, hopefully.

We could keep the public-API-using fuzz tests separate in oss-fuzz, but they wouldn't really like that, and that actually splits up the tests in a confusing way -- some would be here, some would be in the oss-fuzz repo. I think they're better off all being here.

#include <stdlib.h>
#include <inttypes.h>
Expand All @@ -6,7 +7,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);

static PyObject* _fuzz_run(PyObject* self, PyObject* args) {
const char* buf;
size_t size;
Py_ssize_t size;
if (!PyArg_ParseTuple(args, "s#", &buf, &size)) {
return NULL;
}
Expand Down Expand Up @@ -41,7 +42,7 @@ static struct PyModuleDef _fuzzmodule = {
};

PyMODINIT_FUNC
PyInit__fuzz(void)
PyInit__xxtestfuzz(void)
{
PyObject *m = NULL;

Expand Down
File renamed without changes.
File renamed without changes.
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -717,8 +717,8 @@ def detect_modules(self):

# Fuzz tests.
exts.append( Extension(
'_fuzz',
['_fuzz/_fuzzmodule.c', '_fuzz/fuzzer.c'])
'_xxtestfuzz',
['_xxtestfuzz/_xxtestfuzz.c', '_xxtestfuzz/fuzzer.c'])
)

#
Expand Down