Skip to content

defaultdict raises SystemError, __missing__ returned NULL in thread #74658

Closed
Closed
defaultdict raises SystemError, __missing__ returned NULL in thread#74658
Assignees
vstinner
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)type-crashA hard crash of the interpreter, possibly with a core dump

Description

@PeterParente
PeterParente
mannequin
opened on May 25, 2017
BPO 30473
Nosy @vstinner, @serhiy-storchaka, @zhangyangyu, @mlouielu
Superseder
  • bpo-30524: iter(classmethod, sentinel) broken for Argument Clinic class methods?
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vstinner'
closed_at = <Date 2017-06-20.20:34:13.205>
created_at = <Date 2017-05-25.12:21:49.767>
labels = ['interpreter-core', 'type-crash']
title = 'defaultdict raises SystemError, __missing__ returned NULL in thread'
updated_at = <Date 2017-07-01.22:25:16.704>
user = 'https://bugs.python.org/PeterParente'

    bugs.python.org fields:

    activity = <Date 2017-07-01.22:25:16.704>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2017-06-20.20:34:13.205>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2017-05-25.12:21:49.767>
creator = 'Peter Parente'
dependencies = []
files = []
hgrepos = []
issue_num = 30473
keywords = []
message_count = 10.0
messages = ['294473', '294477', '294482', '294484', '294497', '294504', '296485', '296487', '297466', '297498']
nosy_count = 5.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'xiang.zhang', 'louielu', 'Peter Parente']
pr_nums = []
priority = 'critical'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '30524'
type = 'crash'
url = 'https://bugs.python.org/issue30473'
versions = ['Python 3.6']

    Activity

    1. PeterParente commented on May 25, 2017

      @PeterParente
      PeterParentemannequin
      on May 25, 2017
      MannequinAuthor

      Working on this PR (adtech-labs/spylon#49), I encountered an unexpected SystemError in Python 3.6 on my Mac and on Travis Linux.

      Exception in thread Thread-4:
Traceback (most recent call last):
  File "/Users/parente/miniconda3/envs/spylon-dev/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/parente/miniconda3/envs/spylon-dev/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/parente/projects/spylon/spylon/spark/progress.py", line 99, in _spark_progress_thread_worker
    td = datetime.datetime.now() - start_times[stage_id]
SystemError: <built-in method __missing__ of collections.defaultdict object at 0x106013098> returned NULL without setting an error

      The exception only occurs in Python 3.6, not in Python 3.5 or 3.4: https://travis-ci.org/maxpoint/spylon/builds/235992988

      The defaultdict is both created and used as a local variable in a threading.Thread run() function. It's never accessed outside that context (adtech-labs/spylon@2bd47dc#diff-1ba10d1cd92a35a380442cb586e310e2R179)

      I didn't see a related issue on the tracker here nor did I see anything in the 3.6 changelog about a behavior change.

    2. PeterParente
      added
      type-crashA hard crash of the interpreter, possibly with a core dump
      on May 25, 2017

    3. PeterParente commented on May 25, 2017

      @PeterParente
      PeterParentemannequin
      on May 25, 2017
      MannequinAuthor

      Much simpler example: https://gist.github.com/parente/a4772297459f05e43e12a5820051431b

      Wrapping the datetime.datetime.now in a lambda avoids the issue.

    4. mlouielu commented on May 25, 2017

      @mlouielu
      mlouielumannequin
      on May 25, 2017
      Mannequin

      Another way to reproduce this problem:

      >>> import datetime
>>> from collections import defaultdict as dd
>>> d = dd(datetime.datetime.now)
>>> d['foo']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: <built-in method __missing__ of collections.defaultdict object at 0x7f072e55b3b8> returned NULL without setting an error

      This problem doesn't appear on Python 2.7, too.

    5. zhangyangyu commented on May 25, 2017

      @zhangyangyu
      zhangyangyu
      on May 25, 2017
      Member

      Interestingly I could only reproduce the failure on 3.6 but not master.

    6. mlouielu commented on May 25, 2017

      @mlouielu
      mlouielumannequin
      on May 25, 2017
      Mannequin

      This bug is introduce at commit: 37e4ef7, then fixed at commit: 998c209

    7. serhiy-storchaka commented on May 25, 2017

      @serhiy-storchaka
      serhiy-storchaka
      on May 25, 2017
      Member

      In 3.6 _PyStack_UnpackDict() returns just args if there are no keyword arguments. But if args is NULL, the result of _PyStack_UnpackDict() is treated as error.

      Thank you for the reproducer Louie!

    8. serhiy-storchaka
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on May 25, 2017
    9. serhiy-storchaka
      assigned
      vstinner
      on May 25, 2017

    10. vstinner commented on Jun 20, 2017

      @vstinner
      vstinner
      on Jun 20, 2017
      Member

      This bug is a duplicate of bpo-30524.

      Louie Lu: "This bug is introduce at commit: 37e4ef7, then fixed at commit: 998c209"

      This is the fix for master. For Python 3.6, the fix is the commit f0ff849. I added wrote unit tests for this bug, but also for many ways to calls functions: commit b757745.

      Much simpler example: https://gist.github.com/parente/a4772297459f05e43e12a5820051431b

      I'm able to reproduce the bug in Python 3.6 before f0ff849, and also confirm that this commit fixes the bug.

      The good news is that Python 3.6.2rc1 is already available and contains my fix, and 3.6.2 final should be released soon.

    11. vstinner
      closed this as completedon Jun 20, 2017

    12. vstinner commented on Jun 20, 2017

      @vstinner
      vstinner
      on Jun 20, 2017
      Member

      Oh, and thank you Peter Parente for the bug report, and for the script to reproduce the bug. It was usual to validate that the bug is now fixed.

    13. PeterParente commented on Jul 1, 2017

      @PeterParente
      PeterParentemannequin
      on Jul 1, 2017
      MannequinAuthor

      Glad to hear it. Cheers, Victor!

    14. vstinner commented on Jul 1, 2017

      @vstinner
      vstinner
      on Jul 1, 2017
      Member

      It was usual to validate that the bug is now fixed.

      Crap, I wanted to write: useful, not usual!?

    15. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions