tempfile doesn't seem to play nicely with os.chdir on Windows #86962

Activity

1. P403n1x87 commented on Dec 31, 2020

   

   P403n1x87mannequin

   on Dec 31, 2020

   MannequinAuthor

   More actions

   The following script causes havoc on Windows while it works as expected on Linux

   import os
   import tempfile
   
   
   ```py
   def test_chdir():
       with tempfile.TemporaryDirectory() as tempdir:
           os.chdir(tempdir)

   
   
   Running the above on Windows results in RecursionError: maximum recursion depth exceeded while calling a Python object (see attachment for the full stacktrace).
   

   👍React with 👍1Reacted by Saniya Maheshwari
2. P403n1x87

   added

   3.9 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Dec 31, 2020
3. tiran commented on Jan 1, 2021

   

   tiran

   on Jan 1, 2021

   Member

   More actions

   The code fails because TemporaryDirectory.__exit__() is unable to remove the directory. Windows doesn't let you remove files and directories that are used by a process. On POSIX-like operating systems like Linux support removing of opened files. For example anonymous temporary files use the trick.
4. P403n1x87 commented on Jan 1, 2021

   

   P403n1x87mannequin

   on Jan 1, 2021

   MannequinAuthor

   More actions

   That makes sense, but I wonder what the "right" behaviour should be in this case. Surely the infinite recursion should be fixed at the very minimum. Perhaps the code on Windows could be enhanced to catch the case whereby one is trying to delete the cwd and do something like chdir('..') and then delete the temp folder. However, I suspect that something like this still wouldn't be enough. For example, this works fine

   def test_chdir():
       with tempfile.TemporaryDirectory() as tempdir:
           old = os.getcwd()
           os.chdir(tempdir)
           os.chdir(old)
   \~\~\~
   
   whereas this doesn't (same stacktrace as the original case)
   
   ~~~ python
   def test_chdir():
       with tempfile.TemporaryDirectory() as tempdir:
           old = os.getcwd()
           os.chdir(tempdir)
           with open(os.path.join(tempdir, "delme")) as fout:
               fout.write("Hello")
           os.chdir(old)
   \~\~\~
5. eryksun commented on Jan 1, 2021

   

   eryksun

   on Jan 1, 2021

   Contributor

   More actions

   Windows doesn't let you remove files and directories that are used
   by a process.

   Windows does allow deleting open files/directories, but read/execute, write/append, and delete/rename access have to be explicitly shared when opening a file or directory. WinAPI SetCurrentDirectoryW (called by os.chdir) does not share delete access when opening a directory as the new working directory, and the handle is kept open to use as the NTAPI RootDirectory handle when opening relative paths. So the directory can only be deleted after either the working directory changes or the process exits.

   See bpo-35144 for the source of the recursion error. An onerror handler was added that retries rmtree() after attempting to reset permissions. I proposed a workaround in the linked issue, but I don't think it's enough. Maybe there should be a separate onerror function for Windows.

   The two common PermissionError cases to handle in Windows are readonly files and sharing violations. If the readonly attribute is set, we can remove it and retry the unlink() or rmdir() call. I don't think there's no need for a recursive call since rmtree() only removes empty directories.

   For a sharing violation (winerror 32), all that can be done is to set an atexit function that retries deleting the directory. If rmtree() still fails, emit a resource warning and give up.
6. eryksun commented on Jan 1, 2021

   

   eryksun

   on Jan 1, 2021

   Contributor

   More actions

   def test_chdir():
   with tempfile.TemporaryDirectory() as tempdir:
   old = os.getcwd()
   os.chdir(tempdir)
   with open(os.path.join(tempdir, "delme")) as fout:
   fout.write("Hello")
   os.chdir(old)

   The open() call in test_chdir() fails before os.chdir(old) executes. You would need to call os.chdir(old) in a finally block.

   But the cleanup routine could still fail due to a sharing violation from some other process (likely a child process) setting the directory or a child directory as the working directory. Also, a sharing violation when trying to delete a regular file in the tree causes the cleanup routine to fail with NotADirectoryError because the onerror function calls rmtree() if unlink() fails with a PermissionError.
7. ezio-melotti

   transferred this issue fromon Apr 10, 2022
8. serhiy-storchaka

   added this to Tempfile issueson May 20, 2022
9. kwi-dk

   added a commit that references this issue on Dec 1, 2022

   tempfile.TemporaryDirectory: fix recursion bug (seen in pythongh-79325,

   ...

   7e118a0
10. serhiy-storchaka

    added

    OS-windows

    3.11only security fixesonly security fixes

    3.12only security fixesonly security fixes

    3.13only security fixesonly security fixes

    and removed

    3.9 (EOL)end of lifeend of life

    type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

    on Dec 13, 2023