Tkinter hangs if using multiple threads and event handlers #77593

Activity

1. native-api commented on May 2, 2018

   

   native-apimannequin

   on May 2, 2018

   MannequinAuthor

   More actions

   With threaded Tkinter, TkinterHanders3.py from https://bugs.python.org/issue33257 (attached) hangs.

   Tracing with thread_debug and a modified trace.py (to show TIDs, attached) shows that worker threads are waiting for the Tcl lock while the main thread that holds it keeps waiting for some other lock with a strange timeout:

   19000: PyThread_acquire_lock_timed(00000000001B0F80, 0) called
   19000: PyThread_acquire_lock(00000000001B0F80, 0) -> 0
   19000: PyThread_acquire_lock_timed(00000000001B0F80, -1000000) called

   Tested on 3.6 head, win7 x64, debug build.
2. native-api

   added

   3.7 (EOL)end of lifeend of life

   3.8 (EOL)end of lifeend of life

   topic-tkinter

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on May 2, 2018
3. native-api commented on May 2, 2018

   

   native-apimannequin

   on May 2, 2018

   MannequinAuthor

   More actions

   worker threads are waiting for the Tcl lock

   Pardon. They are waiting for Tkapp_ThreadSend()s into the main thread to return. The effect is still the same.
4. gvanrossum commented on May 2, 2018

   

   gvanrossum

   on May 2, 2018

   Member

   More actions

   So it seems threads and Tkinter events don't mix. This doesn't surprise me much. (Similar issues can occur when mixing threads and asyncio if you don't follow the documentation's advice about how to send events across threads.)

   Perhaps event_generate() needs to be more careful with locking?

   Do you have a suggestion for what to do short of dropping Tkinter support?
5. native-api commented on May 3, 2018

   

   native-apimannequin

   on May 3, 2018

   MannequinAuthor

   More actions

   Do you have a suggestion for what to do short of dropping Tkinter support?

   Didn't really look into this.
   At first glance, from the trace log, the main thread seems to grab a lock at some initial point, and then tries to grab it again when running an event handler. So making the lock reentrant may help.
6. gvanrossum commented on May 3, 2018

   

   gvanrossum

   on May 3, 2018

   Member

   More actions

   I guess nobody gives a damn.
7. gvanrossum

   closed this as completedon May 3, 2018
8. terryjreedy commented on May 3, 2018

   

   terryjreedy

   on May 3, 2018

   Member

   More actions

   From 1994 to 2017, _tkinter.c has received a steady flow of multiple revisions each year, for 333 total, by maybe 20 people including Guido. This makes it one of the more actively maintained files and indicates the opposite of indifference and not caring.

   I tested event generation from threads on my Win 10 machine. With installed 2.7, with non-thread tcl 8.5, the process hangs as described, after 1 key event is sent and received. So we should document "don't do that".

   With installed 64-bit 3.6 with thread-compiled tcl 8.6, I see something completely different. The process runs for 5 seconds until the stop call. The two threads alternate sending key events, which are all received in the main thread. Ditto for built 32-big debug 3.6 and 3.8.

   The only problem is that the first t.join() hangs because of a thread deadlock bug. t.join() blocks until t.run exits. t.run does not exit until the last event_generate, with running=False, returns. But that blocks until dummy_handler runs. Dummy_handler does not run when the main thread is blocked by t.join.

   I fixed this by not generating events when running is False. The revised program exits properly.

       def run(self):
           tid = self.ident
           while True:
               time.sleep(0.02)
               c = random.choice(string.ascii_letters)
               print(running, "%d: sending '%s'"%(tid,c),file=sys.stderr)
               if running:
                   self.target.event_generate(c)
               else:
                   break

   I suppose there is a teeny possibility that 'running' could be flipped between the test and the call. Can that be prevented with a lock?

   Another possibility is for stop() to change conditions so that 'self.target.event_generate(c)' fails with an exception, and change if/else to for/except. My first try sort of works in IDLE but not the console.
9. native-api commented on May 4, 2018

   

   native-apimannequin

   on May 4, 2018

   MannequinAuthor

   More actions

   Another possibility is for stop() to change conditions so that 'self.target.event_generate(c)' fails with an exception

   Could you elaborate? Since there're no docs on event_generate(), I can't look up how to make it "fail with an exception" without actually posting an event.

   The only problem is that the first t.join() hangs because of a thread deadlock bug. t.join() blocks until t.run exits. t.run does not exit until the last event_generate, with running=False, returns. But that blocks until dummy_handler runs.
   I suppose there is a teeny possibility that 'running' could be flipped between the test and the call. Can that be prevented with a lock?

   The idea is to let the worker threads finish their work, not terminate them forcibly.
   So the real problem is that stop() blocks the event loop.
   It should rather run asynchronously, wait for threads, then trigger self.root.destroy() in the main thread... somehow.
10. native-api

    added

    invalid

    on May 4, 2018
11. native-api commented on May 4, 2018

    

    native-apimannequin

    on May 4, 2018

    MannequinAuthor

    More actions

    Attached a fixed script.

    Tk.after() works from a worker thread, while Tk.destroy() doesn't.

    That's because Tkinter implements Tcl calls (_tkinter.c:Tkapp_Call) from another thread by posting an event to the interpreter's queue (Tcl_ThreadQueueEvent) and waiting for result. So a call normally works, but would hang if the interpreter's event loop is not running.

    destroy()'s Python part (Lib\tkinter_init_.py:2055) stops the event loop, then makes more Tcl calls -- which hang for the aforementioned reason if made from another thread.
12. terryjreedy commented on May 6, 2018

    

    terryjreedy

    on May 6, 2018

    Member

    More actions

    When I closed the main thread by clicking [x], thus destroying root, both event threads raised instead of hanging. So my experiment involved calling root.destroy instead of setting running to False. The better result when running under IDLE might be due to IDLE's run process executes tcl.call('update') about 20 times per second. Even if the exception idea could be made to work, it seems like a kludge.

    Waiting on the event threads from a separate non-gui thread and leaving the main thread fully functional and responsive until the gui threads die seems much cleaner. Perhaps this should be recommended as a standard way to shut down the main thread when there might be active gui threads. Thank you for following through with this to get a solution we both like.
13. terryjreedy commented on May 6, 2018

    

    terryjreedy

    on May 6, 2018

    Member

    More actions

    Without thread support, event generation from multiple threads fails immediately. I tried an experiment with callback scheduling. It seems to work -- almost.

    thread_event.py runs on 2.7 with non-t tcl. It modifies TkinterHandlres32.py by replacing
    self.target.event_generate(c)
    with
    self.target.after(1, lambda t=self.target: t.event_generate(c))
    to schedule the event generation in the main thread.
    It also imports either tkinter or Tkinter, and runs for 10 seconds
    self.root.after(10000,self.stop)
    for a more rigorous test.

    However, when I add 2 0s to the delay, to make it 1000 seconds, the main thread and gui crash sometime sooner (100 seconds, say), leaving the worker threads sending indefinitely. One time there was a traceback:

    Traceback (most recent call last):
      File "F:\dev\tem\thread_event.py", line 55, in <module>
        Main().go()
      File "F:\dev\tem\thread_event.py", line 35, in go
        self.t_cleanup.join()
    AttributeError: 'Main' object has no attribute 't_cleanup'

    A second time, nothing appeared.

    I suspect that without proper locking an .after call was eventually interrupted and the pending scheduled callback data structure corrupted. Mainloop exits without t_cleanup created.
14. native-api commented on May 6, 2018

    

    native-apimannequin

    on May 6, 2018

    MannequinAuthor

    More actions

    Without thread support, event generation from multiple threads fails immediately.

    This ticket is for threaded Tcl only, so this is off topic.

    In nonthreaded Tcl, this script crashes rather than freezes, for an entire ly different reason that I already explained in https://bugs.python.org/issue33257 .

    This ticket is solved if you ask me.
    The only remaining matter is that there's no documentation:

    • on Tkinter threading model: https://docs.python.org/3/library/tk.html claims full thread safety which is untrue.
    • on best practices with Tkinter: as you could see, all the more or less obvious solutions are flawed. (That includes my solution: the program doesn't terminate gracefully if you close the window by hand.)

    I'm going to cover at least the first item as part of executing Guido's suggestion to "add a warning to the docs".
15. ezio-melotti

    transferred this issue fromon Apr 10, 2022