TST: rare flakyness (or pollution) in a recently introduced subprocess-based test

@neutrinoceros

Looking for other sources of test pollution locally, I hit (I assume by chance) a failure mode for this test, introduced in #17598

astropy/astropy/tests/tests/test_imports.py

Lines 46 to 56 in 768c5ff

    
           def test_toplevel_lazy_imports(): 
        
               # Check that subpackages are loaded on demand. 
        
               cmd = dedent(""" 
        
               import astropy, sys 
        
               assert 'astropy.units' not in sys.modules 
        
               astropy.units 
        
               assert 'astropy.units' in sys.modules 
        
               """) 
        
               cp = subprocess.check_call([sys.executable, "-c", cmd]) 
        
               assert cp == 0

It is extremely puzzling to me that this test might fail in a subprocess, let alone have anything to do with pollution, but unfortunately it is also very hard to debug; I don't even have a good traceback to work with. All I have is

subprocess.CalledProcessError: Command '['/Users/clm/dev/astropy-project/coordinated/astropy/.venv/bin/python3', '-c', "\nimport astropy, sys\nassert 'astropy.units' not in sys.modules\nastropy.units\nassert 'astropy.units' in sys.modules\n"]' returned non-zero exit status 1.

I don't have a clear idea how to rewrite this test to at least make it easier to debug when it fails. I tried using monkeypatch.delitem(sys.modules, "astropy.units", raising=False), but it doesn't suffice to simulate "astropy.units has not been imported yet" and the second assert statement fails.

Originally posted by @neutrinoceros in #17598 (comment)

ping @mhvk

Ugh I wonder if we need a new flag to skip this test from double run...

I wish I had any clue -- obviously the whole point of using the subprocess was to ensure that it is just like having a new python process. In principle, I think one can keep the output of subprocess, so that might help in debugging future instances. Or perhaps instead it should call the pytest executable instead so one gets more of a report?

Would something like this be good enough?

pytest-dev/pytest#12188

@neutrinoceros

@neutrinoceros - did you encounter the problem in running tests twice? Or was it more random still?

Yes, I ran the whole test suite twice, specifically without docs, though I doubt that it has anything to do with it. In hindsight, I need to try to reproduce this, maybe it's actually not that random.

Navigation Menu

Repository navigation

TST: rare flakyness (or pollution) in a recently introduced subprocess-based test #17775

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def test_toplevel_lazy_imports():
	# Check that subpackages are loaded on demand.
	cmd = dedent("""
	import astropy, sys
	assert 'astropy.units' not in sys.modules
	astropy.units
	assert 'astropy.units' in sys.modules
	""")
	cp = subprocess.check_call([sys.executable, "-c", cmd])
	assert cp == 0

TST: rare flakyness (or pollution) in a recently introduced subprocess-based test #17775

Description

Activity

pllim commented on Feb 12, 2025

mhvk commented on Feb 12, 2025

pllim commented on Feb 12, 2025

mhvk commented on Feb 12, 2025

neutrinoceros commented on Feb 12, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions