Repository navigation
Submit the re, json, csv, & struct modules to oss-fuzz testing #73691
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-04-02.16:05:39.827>
created_at = <Date 2017-02-08.20:54:21.924>
labels = ['type-security', '3.7']
title = 'Submit the re, json, csv, & struct modules to oss-fuzz testing'
updated_at = <Date 2021-10-06.23:22:19.809>
user = 'https://github.com/gpshead'bugs.python.org fields:
activity = <Date 2021-10-06.23:22:19.809>
actor = 'miss-islington'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-04-02.16:05:39.827>
closer = 'ammar2'
components = []
creation = <Date 2017-02-08.20:54:21.924>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 29505
keywords = ['patch']
message_count = 29.0
messages = ['287363', '287568', '292821', '293263', '293340', '299183', '299207', '301494', '301495', '301530', '301535', '301732', '301759', '301897', '301901', '301918', '301922', '301932', '345039', '345044', '345045', '345304', '345306', '345307', '346916', '346918', '347496', '390076', '403340']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'terry.reedy', 'gregory.p.smith', 'christian.heimes', 'Devin Jeanpierre', 'alex', 'r.david.murray', 'Guido', 'ammar2', 'miss-islington']
pr_nums = ['2878', '3407', '3437', '13907', '13914', '13915', '13991', '14005', '14006', '14255', '14478', '14479', '18679', '28777']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue29505'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']
Activity
gpshead commentedon Sep 9, 2017
i'd rather make this work in oss-fuzz on cpython. can you point me to how oss-fuzz works and what it wants to do so i can better understand what it needs?
it it has an expectation that the thing being fuzzed is a single binary with no data or directory tree layout dependencies that can just be plopped somewhere else is not a great assumption to make.
but environment variables _should_ be able to be set to point the python binary at what it needs if it must work that way.
ssbr commentedon Sep 11, 2017
I don't have any details except for what's in the PR to oss-fuzz (google/oss-fuzz#731) My understanding is matches what you've said so far:
Python is built to one directory (/out/), but then needs to be run from another directory (/out/ is renamed to /foo/bar/baz/out/). We need python to still work. I have no idea how to do this.
The only suggestion on #python-dev IRC was to statically link a libpython.a, but this doesn't avoid needing to import libraries like "encodings" dynamically, so they still need to be locatable on disk.
Is there a way to build python so that it doesn't use absolute paths to everything, and so that the install can be moved at will? Or is there a way to tell it that it was moved at runtime? (I am unconvinced PYTHONPATH is a maintainable solution, if it works at all...)
oss-fuzz is not going to change away from its model (I asked if they could, they said no), so we're stuck with making Python compatible with it one way or another. This is why I am so drawn to running the test internally on Google's infrastructure anyway: we already _did_ all this work already, via hermetic python. Doing it a second time, but worse, seems annoying.
ssbr commentedon Sep 11, 2017
kcc strongly disagrees though. Copying latest comment:
"""
fwiw - I object to us running any of this internally at Google. We need to be part of the main oss-fuzz project pulling from upstream revisions. Doing this testing within our blackhole of internal stuff adds more work for us internally (read: which we're not going to do) and wouldn't provide results feedback to the upstream CPython project in a useful timely manner.
We must figure out how to get this to build and run on the external oss-fuzz infrastructure
"""
gpshead commentedon Sep 11, 2017
misquote. that was me objecting to running it internally. :)
i believe this is solvable, i haven't had time to spend on this part yet.
ssbr commentedon Sep 11, 2017
Oops, so it is. I can't read apparently.
I'll spend my time on making more fuzz tests in the meantime.
bitdancer commentedon Sep 12, 2017
Seems like it ought to be possible to use the same hooks that venv uses to make this work, but I haven't looked at the details of how those work.
gpshead commentedon Jun 8, 2019
New changeset a15a7bc by Gregory P. Smith (Ammar Askar) in branch 'master':
bpo-29505: Fix interpreter in fuzzing targets to be relocatable (GH-13907)
a15a7bc
miss-islington commentedon Jun 8, 2019
New changeset 22b69da by Miss Islington (bot) in branch '3.8':
bpo-29505: Fix interpreter in fuzzing targets to be relocatable (GH-13907)
22b69da
miss-islington commentedon Jun 8, 2019
New changeset 6692d35 by Miss Islington (bot) in branch '3.7':
bpo-29505: Fix interpreter in fuzzing targets to be relocatable (GH-13907)
6692d35
gpshead commentedon Jun 12, 2019
New changeset a6e190e by Gregory P. Smith (Ammar Askar) in branch 'master':
bpo-29505: Fuzz json module, enforce size limit on int(x) fuzz (GH-13991)
a6e190e
miss-islington commentedon Jun 12, 2019
New changeset 534136a by Miss Islington (bot) in branch '3.7':
bpo-29505: Fuzz json module, enforce size limit on int(x) fuzz (GH-13991)
534136a
miss-islington commentedon Jun 12, 2019
New changeset 878227e by Miss Islington (bot) in branch '3.8':
bpo-29505: Fuzz json module, enforce size limit on int(x) fuzz (GH-13991)
878227e
gpshead commentedon Jun 30, 2019
New changeset 5cbbbd7 by Gregory P. Smith (Ammar Askar) in branch 'master':
bpo-29505: Add more fuzzing for re.compile, re.load and csv.reader (GH-14255)
5cbbbd7
miss-islington commentedon Jun 30, 2019
New changeset ffcc161 by Miss Islington (bot) in branch '3.8':
bpo-29505: Add more fuzzing for re.compile, re.load and csv.reader (GH-14255)
ffcc161
Guido commentedon Jul 8, 2019
Hi,
I've built a generic Python fuzzer and submitted it to OSS-Fuzz.
It works by implementing a "def FuzzerRunOne(FuzzerInput):" function in Python in which some arbitrary code is run based on FuzzerInput, which is a bytes object.
This is a more versatile solution than the current re, json, csv fuzzers as it requires no custom C code and adding more fuzzing targets is as easy as writing a new harness in Python and adding a build rule.
Code coverage is measured at both the CPython level (.c) and the Python level (.py). CPython is compiled with AddressSanitizer. What this means is that both CPython memory bugs and Python library bugs (excessive memory consumption, hangs, slowdowns, unexpected exceptions) are expected to transpire.
You can see my current set of fuzzers here: https://github.com/guidovranken/python-library-fuzzers
The PR to OSS-Fuzz is google/oss-fuzz#2567
Currently, the only Python maintainer who will be receiving automated bug reports is gpshead. Are there any other developers who normally process Python security bug reports and would like to receive notifications?
Feel free to respond directly in the OSS-Fuzz PR thread.
[-]Submit the re, json, & csv modules to oss-fuzz testing[/-][+]Submit the re, json, csv, & struct modules to oss-fuzz testing[/+]ammaraskar commentedon Apr 2, 2021
All the modules prescribed in the original bug are now actively being fuzzed, thank you to Devin for the original work on this and Gregory for reviewing all these changes.
I'm closing this now, feel free to make a new bug and nosy me in if there are any other C-based modules commonly exposed to user input that should be fuzzed.
miss-islington commentedon Oct 6, 2021
New changeset db72e58 by Ammar Askar in branch 'main':
bpo-29505: Add fuzzer for ast.literal_eval (GH-28777)
db72e58
gh-73691: Increase size limits in _xxtestfuzz (#99070)
gpshead commentedon Nov 24, 2024
In anybody has a clone of the, now former https://github.com/guidovranken/python-library-fuzzers repo. Please let us know. The author has deleted their project.
alex commentedon Nov 24, 2024
@gpshead has anyone emailed Guido? If not, I'm happy to.
gpshead commentedon Nov 24, 2024
I emailed them earlier today. But we've since turned up copies so I think we're all good now to figure out if we want to keep running that.