Unable to restart train.py after crash due to thread not being closed properly #1

Activity

1. cmaclell

   added

   bugSomething isn't workingSomething isn't working

   on Jul 13, 2019
2. cmaclell commented on Jul 13, 2019

   

   cmaclell

   on Jul 13, 2019

   Last edited by cmaclell

   CollaboratorAuthor

   More actions

   Currently processes are killed using the atexit module, which runs registered functions when the python program is terminated. However, it has the following edge case:

   Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.
3. cmaclell commented on Jul 13, 2019

   

   cmaclell

   on Jul 13, 2019

   CollaboratorAuthor

   More actions

   An alternative suggested on stack overflow (https://stackoverflow.com/questions/930519/how-to-run-one-last-function-before-getting-killed-in-python) is to use the signal package.

   from signal import *
   import sys, time
   
   def clean(*args):
       print "clean me"
       sys.exit(0)
   
   for sig in (SIGABRT, SIGBREAK, SIGILL, SIGINT, SIGSEGV, SIGTERM):
       signal(sig, clean)
   
   time.sleep(10)
   
4. cmaclell commented on Jul 14, 2019

   

   cmaclell

   on Jul 14, 2019

   CollaboratorAuthor

   More actions

   So I tried replacing atexit with signal as specified above and it did not seem to fix the problem.

   I'm not really sure what is causing the processes to stay up and the port to be blocked.
5. DannyWeitekamp commented on Jul 14, 2019

   

   DannyWeitekamp

   on Jul 14, 2019

   Collaborator

   More actions

   My solution to this is to leave the Port id blank in the net.conf. That will force it to search for an open port. Unfortunately this solution leaves a bunch of running agents which need to be killed. The issue is that django doesn’t want to die. For the host server I have an explicit QUIT request, but there is no such thing in django, because obviously the client shouldn’t be able to kill the server. Unfortunately it also doesn’t seem to respect sig TERM calls which is really annoying. I’ve been trying to figure out how to do this right for a while.
6. eharpste commented on Jun 5, 2020

   

   eharpste

   on Jun 5, 2020

   Member

   More actions

   @cmaclell is this still an issue?