test_wmi.test_wmi_query_threads() crashed with Windows fatal exception: code 0xc0020043 (RPC_NT_INTERNAL_ERROR) on AMD64 Windows PGO Tailcall 3.x #159118

Activity

1. vstinner

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   testsTests in the Lib/test dirTests in the Lib/test dir

   OS-windows

   on Oct 10, 2026
2. vstinner

   changed the title [-]test_wmi.test_wmi_query_threads() crashed with Windows fatal exception: code 0xc0020043 on AMD64 Windows PGO Tailcall 3.x[/-] [+]test_wmi.test_wmi_query_threads() crashed with Windows fatal exception: code 0xc0020043 (RPC_NT_INTERNAL_ERROR) on AMD64 Windows PGO Tailcall 3.x[/+] on Oct 10, 2026
3. vstinner commented on Oct 10, 2026

   

   vstinner

   on Oct 10, 2026

   MemberAuthor

   More actions

   fatal exception: code 0xc0020043

   It's the RPC_NT_INTERNAL_ERROR.
4. utam-1 commented on Oct 13, 2026

   

   utam-1

   on Oct 13, 2026

   More actions

   Hello @vstinner , I was able to partially reproduce this behaviour, please have a look (Windows 11, Version: 23H2):
   This was achieved by running concurrent queries under high CPU load (trying to simulate Buildbot's behaviour)

   Starting WMI test under CPU load...
   Process 0: 148 errors: {'TimeoutError: [WinError 258] The wait operation timed out'}
   Process 1: 150 errors: {'TimeoutError: [WinError 258] The wait operation timed out'}
   Process 2: 148 errors: {'TimeoutError: [WinError 258] The wait operation timed out'}
   Process 3: 147 errors: {'TimeoutError: [WinError 258] The wait operation timed out'}
   Done!
   Windows Event Log Entry (Microsoft-Windows-WMI-Activity/Operational, Event ID 5858):
   TimeCreated : 13-10-2026 06:44:18
   Id          : 5858
   Level       : Error
   Message     : Id = {00000000-0000-0000-0000-000000000000}; Component = Unknown;
                 Operation = Start IWbemServices::ExecQuery - ROOT\CIMV2 : 
                 SELECT ProcessId FROM Win32_Process WHERE ProcessId < 1000; 
                 ResultCode = 0x80041032; PossibleCause = Unknown
   
   Operation = Start IWbemServices::ExecQuery - ROOT\CIMV2 : ...; ResultCode = 0x80041032 (WBEM_E_CALL_CANCELLED)
   
   

   Based on my analysis of _wmimodule.cpp maybe queries time out and leave runaway background threads behind. Those threads can later mess with reused handles, and with no error-catching, the program crashes. ( _wmi_exec_query_impl, wait_event(data.connectEvent, 100)), so maybe it is plausible to relax this timeout?

   I'd like to work on this(and open up a PR if possible), but would like to discuss the approach first with you so that we can figure out what to change.

   Disclosure: I have taken help from AI agents to research on this issue.
5. vstinner commented on Oct 13, 2026

   

   vstinner

   on Oct 13, 2026

   MemberAuthor

   More actions

   Can you share the script that you used to reproduce the issue?
6. utam-1 commented on Oct 13, 2026

   

   utam-1

   on Oct 13, 2026

   More actions

   5. Can you share the script that you used to reproduce the issue?

   This is the script I've used, if I'm running it again I have got kind of similar output related to RPC_NT_INTERNAL_ERROR.

   import concurrent.futures
   import multiprocessing
   import _wmi
   
   def run_wmi_queries(iterations):
       wmi_query = "SELECT ProcessId FROM Win32_Process WHERE ProcessId < 1000"
       failures = []
       
       for _ in range(iterations):
           try:
               response = _wmi.exec_query(wmi_query)
               if not response or "ProcessId=" not in response:
                   failures.append("invalid_response")
           except Exception as err:
               failures.append(f"{type(err).__name__}: {err}")
               
       return failures
   
   def run_threaded_batch(worker_count, iterations_per_worker):
       with concurrent.futures.ThreadPoolExecutor(max_workers=worker_count) as thread_pool:
           jobs = [
               thread_pool.submit(run_wmi_queries, iterations_per_worker) 
               for _ in range(worker_count)
           ]
           batch_errors = []
           for job in concurrent.futures.as_completed(jobs):
               batch_errors.extend(job.result())
               
       return batch_errors
   
   def load_generator(cancel_flag):
       while not cancel_flag.is_set():
           _ = [val * val for val in range(10000)]
   
   if __name__ == '__main__':
       shutdown_signal = multiprocessing.Event()
       load_workers = [
           multiprocessing.Process(target=load_generator, args=(shutdown_signal,)) 
           for _ in range(4)
       ]
       
       for worker in load_workers:
           worker.start()
           
       try:
           with concurrent.futures.ProcessPoolExecutor(max_workers=4) as process_pool:
               tasks = [process_pool.submit(run_threaded_batch, 8, 20) for _ in range(4)]
               
               for idx, task in enumerate(tasks):
                   errors = task.result()
                   print(f"Worker {idx}: {len(errors)} errors encountered: {set(errors)}")
       finally:
           shutdown_signal.set()
           for worker in load_workers:
               worker.join()
   

   OUTPUT
   ----------
   Worker 0: 137 errors encountered: {'TimeoutError: [WinError 258] The wait operation timed out'}
   Worker 1: 143 errors encountered: {'TimeoutError: [WinError 258] The wait operation timed out'}
   concurrent.futures.process._RemoteTraceback:
   '''
   Process 8744 terminated abruptly with exit code 3221356611'''
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "...\repro_wmi_v2.py", line 50, in <module>
       errors = task.result()
     File "C:\Users\user\Desktop\cpython\Lib\concurrent\futures\_base.py", line 461, in result
       return self.__get_result()
              ~~~~~~~~~~~~~~~~~^^
     File "C:\Users\user\Desktop\cpython\Lib\concurrent\futures\_base.py", line 403, in __get_result
       raise self._exception
   concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
   
7. vstinner commented on Oct 14, 2026

   

   vstinner

   on Oct 14, 2026

   MemberAuthor

   More actions

   Another error on AMD64 Windows10 3.x: https://buildbot.python.org/#/builders/146/builds/14554

   ERROR: test_wmi_query_repeated (test.test_wmi.WmiTests.test_wmi_query_repeated)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_wmi.py", line 41, in test_wmi_query_repeated
       self.test_wmi_query_os_version()
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
     File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_wmi.py", line 29, in test_wmi_query_os_version
       r = wmi_exec_query("SELECT Version FROM Win32_OperatingSystem").split("\0")
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_wmi.py", line 17, in wmi_exec_query
       return _wmi.exec_query(query)
              ~~~~~~~~~~~~~~~^^^^^^^
   OSError: [WinError -2147023489] Cannot find window class
   

   The test passed when re-run in verbose mode:

   test_wmi_query_repeated (test.test_wmi.WmiTests.test_wmi_query_repeated) ... ok
   
8. utam-1 commented on Oct 15, 2026

   

   utam-1

   on Oct 15, 2026

   More actions

   Hello @vstinner, I investigated this issue further, check PC/_wmimodule.cpp, hr = CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED);. I've looked into the STA architecture, it creates a hidden window class to communicate across COM. So every time Python ran a query, it : "made a new thread, built a hidden window for it (STA), ran the query, tore everything down". Doing this over and over, especially when the computer is very busy, causes trouble. Windows sometimes fails to register or find its hidden window, giving the “Cannot find window class” error. And when many STA threads waited on the background service without a working mailbox loop, the communication broke, and the process crashed.

   When I changed: hr = CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED);, to hr = CoInitializeEx(nullptr, COINIT_MULTITHREADED); and ran my test script that I mentioned above the issue was gone and I couldn't see any crashes.

   Would you please test this in the actual environment, and confirm if this works for you?
9. zooba commented on Oct 15, 2026

   

   zooba

   on Oct 15, 2026

   Member

   More actions

   Switching to MTA sounds reasonable. I'm not super familiar with the internals, but the CoInitializeEx docs do make it sound like STA is going to spin up a lot more infrastructure that MTA defers to the COM object implementation. Since we aren't doing threads here anyway, and the WMI samples use MTA, it should be safe enough.