Repository navigation
Non-deterministic bytecode generation for frozenset #78903
Description
| BPO | 34722 |
|---|---|
| Nosy | @rhettinger, @benjaminp, @serhiy-storchaka, @yan12125, @peterebden, @pablogsal, @jefferyto |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-09-18.13:53:47.451>
labels = ['interpreter-core', '3.9']
title = 'Non-deterministic bytecode generation'
updated_at = <Date 2021-09-05.02:16:59.920>
user = 'https://github.com/peterebden'bugs.python.org fields:
activity = <Date 2021-09-05.02:16:59.920>
actor = 'yan12125'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2018-09-18.13:53:47.451>
creator = 'Peter Ebden'
dependencies = []
files = []
hgrepos = []
issue_num = 34722
keywords = ['patch']
message_count = 8.0
messages = ['325644', '325657', '325658', '325736', '325837', '344238', '347820', '401066']
nosy_count = 7.0
nosy_names = ['rhettinger', 'benjamin.peterson', 'serhiy.storchaka', 'yan12125', 'Peter Ebden', 'pablogsal', 'jefferyto']
pr_nums = ['9472']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue34722'
versions = ['Python 3.9']
Activity
peterebden commentedon Sep 18, 2018
We've found that the following code produces non-deterministic bytecode,
even post PEP-552:
It's not too hard to test it:
It looks like the peephole optimiser is converting the LOAD_CONST instructions
for the set into a single LOAD_CONST for a frozenset which then serialises in
nondeterministic order. One can hence work around it by setting PYTHONHASHSEED
to a known value.
I'm happy to help out with this if needed, although I don't have a lot of
familiarity with the relevant code.
rhettinger commentedon Sep 18, 2018
Have a look at line 512 in Python/marshal.c which calls PyObject_GetIter(). We would need to add PySequence_List() and PyList_Sort(). This will slow down marshaling but would make the bytecode deterministic.
serhiy-storchaka commentedon Sep 18, 2018
Not all types are orderable.
peterebden commentedon Sep 19, 2018
Thanks for the pointer, I'll have a bit more of a dig into it (although Serhiy makes a good point too...).
benjaminp commentedon Sep 20, 2018
Possibly we should just sort the individual marsahalled entries of the frozenset.
rhettinger commentedon Jun 1, 2019
Bumping up the priority a bit on this one. It would be nice to get it in for 3.8.
yan12125 commentedon Jul 13, 2019
I encounter another case that leads to non-deterministic bytecode. For example, with commit e6b46aa + PR 9472, I got:
In this case there are no sets. I guess the cause is different. Should I open a new issue?
yan12125 commentedon Sep 5, 2021
bpo-37596 merged a fix to enable deterministic frozensets. I think this issue can be closed?
Regarding my last comment msg347820 - it seems similar to one of https://bugs.python.org/issue34033 or https://bugs.python.org/issue34093. I followed those tickets instead.
[-]Non-deterministic bytecode generation[/-][+]Non-deterministic bytecode generation for frozenset[/+]pythongh-106287: do not write objects after encountering an unmarshal…