Skip to content

Pickle crashes unpickling invalid NEWOBJ_EX opcode #85460

Closed
Closed
Pickle crashes unpickling invalid NEWOBJ_EX opcode#85460
Assignees
serhiy-storchaka
Labels
3.10 (EOL)end of life3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeextension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@ImanSharafaldin
ImanSharafaldin
mannequin
opened on Jul 12, 2020
BPO 41288
Nosy @abalkin, @tiran, @ned-deily, @ambv, @serhiy-storchaka, @pganssle, @miss-islington
PRs
  • bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. #21458
  • [3.9] bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) #21459
  • [3.8] bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) #21460
  • [3.7] bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) #21461
  • [3.6] bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) #21462
  • bpo-41288: Refactor of unpickling NEWOBJ and NEWOBJ_EX opcodes. #21472
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2020-07-26.14:19:49.073>
created_at = <Date 2020-07-12.21:36:09.137>
labels = ['3.8', '3.9', '3.10', 'extension-modules', '3.7', 'type-crash']
title = 'Pickle crashes unpickling invalid NEWOBJ_EX opcode'
updated_at = <Date 2020-07-26.15:12:33.081>
user = 'https://bugs.python.org/ImanSharafaldin'

    bugs.python.org fields:

    activity = <Date 2020-07-26.15:12:33.081>
actor = 'Iman Sharafaldin'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2020-07-26.14:19:49.073>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2020-07-12.21:36:09.137>
creator = 'Iman Sharafaldin'
dependencies = []
files = []
hgrepos = []
issue_num = 41288
keywords = ['patch']
message_count = 15.0
messages = ['373573', '373575', '373599', '373600', '373603', '373604', '373605', '373616', '373617', '373886', '374022', '374086', '374311', '374314', '374320']
nosy_count = 8.0
nosy_names = ['belopolsky', 'christian.heimes', 'ned.deily', 'lukasz.langa', 'serhiy.storchaka', 'p-ganssle', 'miss-islington', 'Iman Sharafaldin']
pr_nums = ['21458', '21459', '21460', '21461', '21462', '21472']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue41288'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

    Activity

    1. ImanSharafaldin commented on Jul 12, 2020

      @ImanSharafaldin
      ImanSharafaldinmannequin
      on Jul 12, 2020
      MannequinAuthor

      The following code generates a segfault on the Pickle module [it's a crafted datetime object] (Python 3.10.0a0 (heads/master:b40e434, Jul 4 2020), Python 3.6.11 and Python 3.7.2):

      import io
import pickle


hex_string = "8004952A000000000000008C086461746574696D65948C086461746574696D65949388430A07B2010100000000000092059452942E"
myb = bytes.fromhex(hex_string)
f = io.BytesIO(myb)
print(f)
data = pickle.load(f)
print(data)
print('We have segfault but we cannot see!')
    2. ImanSharafaldin
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      3.10 (EOL)end of life
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Jul 12, 2020

    3. tiran commented on Jul 12, 2020

      @tiran
      tiran
      on Jul 12, 2020
      Member

      datetime_new assumes that args is a tuple. load_newobj_ex() doesn't check that args is a tuple and kwargs is a dictionary. The demo exploit passes True as args, which triggers a segfault in PyTuple_GET_SIZE in datetime_new.

      #0 datetime_new (type=0x7fffea5d2740 <PyDateTime_DateTimeType>, args=True, kw=b'\x07\xb2\x01\x01\x00\x00\x00\x00\x00\x00') at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_datetimemodule.c:4737
      #1 0x00007fffea637b1e in load_newobj_ex (self=0x7fffea7a7820) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:6008
      #2 0x00007fffea632e7a in load (self=0x7fffea7a7820) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:6943
      #3 0x00007fffea63795e in _pickle_load_impl (module=<optimized out>, buffers=0x0, errors=0x7fffea639149 "strict", encoding=0x7fffea6391fa "ASCII", fix_imports=1, file=<_io.BytesIO at remote 0x7fffea888180>)
      at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:1688
      #4 _pickle_load (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/clinic/_pickle.c.h:731
      #5 0x00007ffff7bd1ced in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method load of module object at remote 0x7fffea680270>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0)
      at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Objects/methodobject.c:437

    4. serhiy-storchaka
      added
      extension-modulesC modules in the Modules dir
      and removed
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on Jul 12, 2020
    5. serhiy-storchaka
      self-assigned this
      on Jul 12, 2020
    6. serhiy-storchaka
      added
      extension-modulesC modules in the Modules dir
      and removed
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on Jul 12, 2020
    7. serhiy-storchaka
      self-assigned this
      on Jul 12, 2020
    8. serhiy-storchaka
      changed the title [-]Pickle crashes using a crafted datetime object[/-] [+]Pickle crashes unpickling invalig NEWOBJ_EX opcode[/+] on Jul 13, 2020

    9. 14 remaining items

    10. ned-deily commented on Jul 13, 2020

      @ned-deily
      ned-deily
      on Jul 13, 2020
      Member

      New changeset 620e276 by Miss Islington (bot) in branch '3.7':
      bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) (GH-21461)
      620e276

    11. ned-deily commented on Jul 13, 2020

      @ned-deily
      ned-deily
      on Jul 13, 2020
      Member

      New changeset 6463cf0 by Miss Islington (bot) in branch '3.6':
      bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) (GH-21462)
      6463cf0

    12. ned-deily
      closed this as completedon Jul 13, 2020
    13. ned-deily
      closed this as completedon Jul 13, 2020

    14. serhiy-storchaka commented on Jul 18, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 18, 2020
      Member

      New changeset b4c98ed by Serhiy Storchaka in branch 'master':
      bpo-41288: Refactor of unpickling NEWOBJ and NEWOBJ_EX opcodes. (GH-21472)
      b4c98ed

    15. ambv commented on Jul 20, 2020

      @ambv
      ambv
      on Jul 20, 2020
      Contributor

      Is this fix causing the refleak build to fail on 3.8? See: https://buildbot.python.org/all/#/builders/484/builds/149

    16. serhiy-storchaka
      reopened this on Jul 21, 2020
    17. serhiy-storchaka
      reopened this on Jul 21, 2020

    18. serhiy-storchaka commented on Jul 22, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 22, 2020
      Member

      This fix touched the unpickling code, and the test crashed during pickling, so it is not directly related. But it may be some compiler glitch, when changing one part of code affects compilation of other parts of code. The next run on the buildbot was passed, and there were failures in the past (but logs were not saved), so it may be just unstable test.

    19. ImanSharafaldin commented on Jul 26, 2020

      @ImanSharafaldin
      ImanSharafaldinmannequin
      on Jul 26, 2020
      MannequinAuthor

      @serhiy.storchaka Hi Serhiy, do you count this issue as a security issue? Also, I didn't see that in the latest changelog. Is it now patched on the latest version?

      https://docs.python.org/release/3.8.5/whatsnew/changelog.html

    20. serhiy-storchaka commented on Jul 26, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 26, 2020
      Member

      I do not think it is a security issue. The crash cannot be triggered by the user input unless you accept the pickle data from untrusted sources, but in that case you are in large danger, because you allow executing arbitrary code.

      The changes in this issue just help to debug in some cases when you play with pickle format. In any case thank you for your report.

    21. serhiy-storchaka
      closed this as completedon Jul 26, 2020
    22. serhiy-storchaka
      closed this as completedon Jul 26, 2020

    23. ImanSharafaldin commented on Jul 26, 2020

      @ImanSharafaldin
      ImanSharafaldinmannequin
      on Jul 26, 2020
      MannequinAuthor

      Thank you for patching that.

    24. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.10 (EOL)end of life3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeextension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions