test_tools are leaked

./python -m test -R 3:3 test_tools    
Running Debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/1] test_tools
beginning 6 repetitions 
123456                  
.C:\Users\KIRILL-1\CLionProjects\cpython\Modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set
 MPD_MINALLOC a second time

.C:\Users\KIRILL-1\CLionProjects\cpython\Modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set
 MPD_MINALLOC a second time

.C:\Users\KIRILL-1\CLionProjects\cpython\Modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set
 MPD_MINALLOC a second time

.C:\Users\KIRILL-1\CLionProjects\cpython\Modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set
 MPD_MINALLOC a second time

.C:\Users\KIRILL-1\CLionProjects\cpython\Modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set
 MPD_MINALLOC a second time

.
test_tools leaked [1960, 1956, 1960] references, sum=5876
test_tools leaked [1075, 1073, 1075] memory blocks, sum=3223
test_tools failed (reference leak) in 46.3 sec

== Tests result: FAILURE ==

1 test failed:
    test_tools

Total duration: 46.3 sec
Tests result: FAILURE

Tried on current main

Linked PRs

GH-107458: fix test_tools refleak #107577

@kumaraditya303

Bisected to #107287
cc @kumaraditya303 @CharlieZhao95 @erlend-aasland

I think problem is here:

cpython/Lib/test/support/import_helper.py

Lines 253 to 268 in a24e25d

    
           def modules_cleanup(oldmodules): 
        
               # Encoders/decoders are registered permanently within the internal 
        
               # codec cache. If we destroy the corresponding modules their 
        
               # globals will be set to None which will trip up the cached functions. 
        
               encodings = [(k, v) for k, v in sys.modules.items() 
        
                            if k.startswith('encodings.')] 
        
               sys.modules.clear() 
        
               sys.modules.update(encodings) 
        
               # XXX: This kind of problem can affect more than just encodings. 
        
               # In particular extension modules (such as _ssl) don't cope 
        
               # with reloading properly. Really, test modules should be cleaning 
        
               # out the test specific modules they know they added (ala test_runpy) 
        
               # rather than relying on this function (as test_importhooks and test_pkg 
        
               # do currently). Implicitly imported *real* modules should be left alone 
        
               # (see issue 10556). 
        
               sys.modules.update(oldmodules)

I cannot reproduce it on Ubuntu 22.04. HEAD->6e850c30bb70

I cannot reproduce it on Ubuntu 22.04. HEAD->6e850c30bb70

Problem was introduced later, 6e850c3 was made on 27th July, original problem introduced in a43cc3f (28th July).
Please, update your local main branch and let me know results of refleak tests.

root@codespaces-62bef3 /w/cpython (main)# ./python -m test -R 3:3 test_decimal
0:00:00 load avg: 1.52 Run tests sequentially
0:00:00 load avg: 1.52 [1/1] test_decimal
./Modules/_decimal/libmpdec/context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 29.3 sec
Tests result: SUCCESS

This suggests the problem lies within this test.

Minimal bug reproducer:

import sys
import unittest
import importlib
from test.support import import_helper


class TestDecimalRefleak(unittest.TestCase):
    def test_refleak(self):
        old_modules = import_helper.modules_setup()
        decimal = importlib.import_module("decimal")
        import_helper.modules_cleanup(*old_modules)

Minimal bug reproducer:

import sys
import unittest
import importlib
from test.support import import_helper


class TestDecimalRefleak(unittest.TestCase):
    def test_refleak(self):
        old_modules = import_helper.modules_setup()
        decimal = importlib.import_module("decimal")
        import_helper.modules_cleanup(*old_modules)

If we add this line sys.modules['decimal'] = decimal to the end of test_refleak, problem will gone, but it seems as incorrect solution.

As far as I understand, decimal_free from _decimal.c is not called. Could this be a problem?

If we add this line sys.modules['decimal'] = decimal to the end of test_refleak, problem will gone, but it seems as incorrect solution.

As you said. If you add this statement, _decimal_exec is actually only executed once. (Note that the warning in mpd_setminalloc will not exist)

If we add this line sys.modules['decimal'] = decimal to the end of test_refleak, problem will gone, but it seems as incorrect solution.

As you said. If you add this statement, _decimal_exec is actually only executed once. (Note that the warning in mpd_setminalloc will not exist)

Oh, I suddenly discovered another issue with _socket..

import sys
import unittest
import importlib
from test.support import import_helper


class TestDecimalRefleak(unittest.TestCase):
    def test_refleak(self):
        old_modules = import_helper.modules_setup()
        socket = importlib.import_module("_socket")
        import_helper.modules_cleanup(*old_modules)

Traceback:
./python.exe -m test -R 3:3 test_refleak
0:00:00 load avg: 2.22 Run tests sequentially
0:00:00 load avg: 2.22 [1/1] test_refleak
beginning 6 repetitions
123456
......
test_refleak leaked [219, 219, 219] references, sum=657
test_refleak leaked [144, 144, 144] memory blocks, sum=432
test_refleak failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_refleak

Total duration: 150 ms
Tests result: FAILURE

Above the same problem, socket_exec called multiple times.

More simple reproducer:

import sys
import unittest
import importlib


class TestDecimalRefleak(unittest.TestCase):
    def test_refleak(self):
        importlib.import_module("_decimal")
        sys.modules.pop("_decimal")

GH-107458: fix test_tools refleak (#107577)

Navigation Menu

Repository navigation

test_tools are leaked #107458

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def modules_cleanup(oldmodules):
	# Encoders/decoders are registered permanently within the internal
	# codec cache. If we destroy the corresponding modules their
	# globals will be set to None which will trip up the cached functions.
	encodings = [(k, v) for k, v in sys.modules.items()
	if k.startswith('encodings.')]
	sys.modules.clear()
	sys.modules.update(encodings)
	# XXX: This kind of problem can affect more than just encodings.
	# In particular extension modules (such as _ssl) don't cope
	# with reloading properly. Really, test modules should be cleaning
	# out the test specific modules they know they added (ala test_runpy)
	# rather than relying on this function (as test_importhooks and test_pkg
	# do currently). Implicitly imported real modules should be left alone
	# (see issue 10556).
	sys.modules.update(oldmodules)

test_tools are leaked #107458

Description

Linked PRs

Activity

Eclips4 commented on Jul 30, 2023

Eclips4 commented on Jul 30, 2023

sunmy2019 commented on Jul 30, 2023

Eclips4 commented on Jul 30, 2023

kumaraditya303 commented on Jul 31, 2023

CharlieZhao95 commented on Jul 31, 2023

Eclips4 commented on Jul 31, 2023

Eclips4 commented on Jul 31, 2023

CharlieZhao95 commented on Jul 31, 2023

Eclips4 commented on Jul 31, 2023

Eclips4 commented on Jul 31, 2023

Eclips4 commented on Jul 31, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions