threading.Thread should have way to catch an exception thrown within #80847

Activity

1. TV4Fun commented on Apr 19, 2019

   

   TV4Funmannequin

   on Apr 19, 2019

   MannequinAuthor

   More actions

   This has been commented on numerous times by others (https://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python, http://benno.id.au/blog/2012/10/06/python-thread-exceptions, to name a few), but there is no in-built mechanism in threading to catch an unhandled exception thrown by a thread. The default behavior of dumping to stderr is completely useless for error handling in many scenarios. Solutions do exist, but I have yet to see one that is not exceptionally complicated. It seems like checking for exceptions should be a very basic part of any threading library. The simplest solution would be to just have the Thread store any unhandled exceptions and have them raised by Thread.join(). There could also be additional methods to check if exceptions were raised.
2. TV4Fun

   added

   3.7 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Apr 19, 2019
3. pitrou

   added

   3.8 (EOL)end of lifeend of life

   type-featureA feature request or enhancementA feature request or enhancement

   and removed

   3.7 (EOL)end of lifeend of life

   on Apr 19, 2019
4. pitrou commented on Apr 19, 2019

   

   pitrou

   on Apr 19, 2019

   Member

   More actions

   The current behavior can't be changed for compatibility reasons (imagine user programs starting to raise on Thread.join()), but we could add an option to the threading.Thread() constructor in order to store and propagate exceptions.
5. TV4Fun commented on Apr 19, 2019

   

   TV4Funmannequin

   on Apr 19, 2019

   MannequinAuthor

   More actions

   I agree that we should not change the default behavior of Thread.join(), as that would break existing code, but there are plenty of other ways to do this. I see a couple of possibilities:

   1. Add an option to the Thread constructor, something like raise_exc, that defaults to False, but when set to True, causes join() to raise any exceptions.

   2. (Better, IMO) Add this option to the join() method instead.

   3. Create a new method, join_with_exc(), that acts like join() but raises exceptions from the target.

   4. (Should probably do this anyway, regardless of what else we do) Add a new method, check_exc(), that checks if any unhandled exceptions have occurred in the thread and returns and/or raises any that have.
6. ericsnowcurrently commented on Apr 19, 2019

   

   ericsnowcurrently

   on Apr 19, 2019

   Member

   More actions

   Here's a basic decorator along those lines, similar to one that I've used on occasion:

       def as_thread(target):
           def _target():
               try:
                   t.result = target()
               except Exception as exc:
                   t.failure = exc
           t = threading.Thread(target=_target)
           return t

   Sure, it's border-line non-trivial, but I'd hardly call it "exceptionally complicated".

   Variations for more flexibility:

       def as_thread(target=None, **tkwds):
           # A decorator to create a one-off thread from a function.
           if target is None:
               # Used as a decorator factory
               return lambda target: as_thread(target, **tkwds)
   
           def _target(*args, **kwargs):
               try:
                   t.result = target(*args, **kwargs)
               except Exception as exc:
                   t.failure = exc
           t = threading.Thread(target=_target, **tkwds)
           return t
   
   
       def threaded(target, **tkwds):
           # A decorator to produce a started thread when the "function" is called.
           if target is None:
               # Used as a decorator factory
               return lambda target: as_thread(target, **tkwds)
   
           @functools.wraps(target)
           def wrapper(*targs, **tkwargs)
               def _target(*args, *kwargs):
                   try:
                       t.result = target(*args, **kwargs)
                   except Exception as exc:
                       t.failure = exc
               t = threading.Thread(target=_target, args=targs, kwargs=tkwargs, **tkwds)
               t.start()
               return t
           return wrapper
7. TV4Fun commented on Apr 19, 2019

   

   TV4Funmannequin

   on Apr 19, 2019

   MannequinAuthor

   More actions

   Yes, I know there are workarounds for it, I have seen many, and everyone seems to have their own version. I'm saying we shouldn't need workarounds though–this should be built in functionality. Ideally, dropping an exception should never be default behavior, but I understand not wanting to break existing code, that's why I'm saying add additional functionality to make these checks easier and not require hacky, un-pythonic wrappers and other methods to find out if your code actually worked.
8. MarkBorgerding commented on Oct 29, 2019

   

   MarkBorgerdingmannequin

   on Oct 29, 2019

   Mannequin

   More actions

   @pitrou I don't necessarily agree that "current behavior can't be changed". One major selling point of exceptions is that they cannot be accidentally ignored. The exception is how the current threading.Thread ignores them.

   You are correct that changing Thread.join() so it propagates exceptions by default may break code that relies on the implicit behavior of a thread dying when the target/run method raises. I'd argue such code deserves to be broken -- "explicit is better than implicit".

   I suspect there is more code that will be fixed by such a change than broken.
9. rhettinger commented on Oct 29, 2019

   

   rhettinger

   on Oct 29, 2019

   Contributor

   More actions

   I'd argue such code deserves to be broken

   That argument falls flat with me and it doesn't show respect for our users. The proposal would be a major change to the "rules of the game" and would likely require discussion and buy-in on python-dev and perhaps a PEP.

   Please heed Antoine's advice. This is his area of expertise, and he has suggested a plausible way to go forward.
10. MarkBorgerding commented on Oct 29, 2019

    

    MarkBorgerdingmannequin

    on Oct 29, 2019

    Mannequin

    More actions

    I'm not trying to disrespect anyone: not users nor certainly Python developers. I have loved Python since I learned it in 2001. I was merely trying to respond to what seemed like an automatic rejection of changing legacy behavior. I certainly agree changes to default behavior should not be made lightly, but sometimes they *should* be made.

    This may be such a case:
    a) A user writing a thread function either consciously thinks about exceptions while they write function or they do not. Sure, we should always do the former; just like we should always eat our veggies and get lots of exercise, but ....
    b) If they *do* think about exceptions, they will either
    b.1) write exception handlers within that function, or
    b.2) rely on default behavior that simply prints out the exception string to stderr and lets the thread quietly die.
    c) If they did not explicitly think about exceptions while they wrote the function, they may have in the back of their mind the reasonable expectation that an exception will not be ignored.

    Changing join() to propagate unhandled exceptions would admittedly break b.2 code, but provides absolution for all sinners who have written case c.

    This is what I meant by, "I suspect there is more code that will be fixed by such a change than broken."

    I'll confess I only just now became aware of threading.excepthook added in python 3.8. Does it change this problem?
11. TV4Fun commented on Oct 29, 2019

    

    TV4Funmannequin

    on Oct 29, 2019

    MannequinAuthor

    More actions

    I'm kind of in agreement with Mark on this, actually. I came across this problem when examining some threaded code that was clearly not working as intended, but was reporting success. Figuring out why that was was not easy. The code had been hastily ported to a multi-threaded version from an iterative version by my predecessor, and neither of us had enough familiarity with Python threads to realize what the problem was. The whole point of having exceptions is that it gives you a way of knowing when errors happen without having to add a bunch of extra error checking to your own code. It rather defeats the purpose if code can silently fail while still throwing exceptions, and we have to add extra code to handle special cases like this where they are ignored.
12. vstinner commented on Sep 22, 2020

    

    vstinner

    on Sep 22, 2020

    Member

    More actions

    I consider that this issue as a duplicate of bpo-1230540. Python 3.8 has a new threading.excepthook hook which can be used in various ways to decide how to handle uncatched thread exceptions.

    https://docs.python.org/dev/library/threading.html#threading.excepthook
13. vstinner

    closed this as completedon Sep 22, 2020
14. ezio-melotti

    transferred this issue fromon Apr 10, 2022