Repository navigation
test_eintr fails on AIX since fcntl functions were modified #79814
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-02-14.18:25:02.289>
created_at = <Date 2019-01-01.18:09:08.290>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_eintr fails on AIX since fcntl functions were modified'
updated_at = <Date 2021-05-10.21:30:57.620>
user = 'https://github.com/aixtools'bugs.python.org fields:
activity = <Date 2021-05-10.21:30:57.620>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-14.18:25:02.289>
closer = 'Mariatta'
components = ['Tests']
creation = <Date 2019-01-01.18:09:08.290>
creator = 'Michael.Felt'
dependencies = []
files = []
hgrepos = []
issue_num = 35633
keywords = ['patch', 'patch']
message_count = 18.0
messages = ['332846', '332936', '333015', '333020', '333048', '333064', '333157', '333179', '333658', '333662', '334021', '334147', '335550', '335552', '335553', '337630', '337701', '393161']
nosy_count = 4.0
nosy_names = ['Michael.Felt', 'Mariatta', 'miss-islington', 'kadler']
pr_nums = ['11424', '11424', '11858', '11500']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35633'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
aixtools commentedon Jan 1, 2019
test_eintr fails on AIX since fcntl functions were modified
In bpo-35189 the fnctl() module was modified so that the EINTR interruption should be retried automatically.
On AIX the test for flock() passes, but the test for lockf() fails:
======================================================================
Researching...
aixtools commentedon Jan 3, 2019
After reading the PEP I realized it is much simpler. The test is for interrupts that occur at a low-level - and not for permission issues. The test is failing because there is a permission issue, not a missed interrupt issue. Modifying the code to: (see line 510)
+506 try:
+507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
+508 lock_func(f, fcntl.LOCK_UN)
+509 time.sleep(0.01)
+510 except (BlockingIOError, PermissionError):
+511 break
+512 # the child locked the file just a moment ago for 'sleep_time' seconds
+513 # that means that the lock below will block for 'sleep_time' minus some
+514 # potential context switch delay
+515 lock_func(f, fcntl.LOCK_EX)
+516 dt = time.monotonic() - start_time
+517 self.assertGreaterEqual(dt, self.sleep_time)
+518 self.stop_alarm()
fixes this.
vstinner commentedon Jan 4, 2019
Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode?
I'm talking about this line:
Note: if you want to test, you have the modify the mode twice:
"with open('%s', 'wb') as f:" % support.TESTFN,
and
with open(support.TESTFN, 'wb') as f:
[-]test_eintr fails on AIX since fcntl functions were modified[/-][+]test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX[/+]vstinner commentedon Jan 4, 2019
Previous discussion at:
https://bugs.python.org/issue35189#msg332580
aixtools commentedon Jan 5, 2019
I had tried “wb+”, not “w+b”. Is there a difference? I forget if I tried just “w+”.
But I’ll do them anyway/again to be sure.
Sent from my iPhone
aixtools commentedon Jan 5, 2019
On 04/01/2019 23:42, STINNER Victor wrote:
Without except containing PermissionError:
+500 with kill_on_error(proc):
+501 with open(support.TESTFN, 'w+b') as f:
+502 while True: # synchronize the subprocess
+503 dt = time.monotonic() - start_time
+504 if dt > 60.0:
+505 raise Exception("failed to sync child in
%.1f sec" % dt)
+506 try:
+507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
+508 lock_func(f, fcntl.LOCK_UN)
+509 time.sleep(0.01)
+510 except BlockingIOError:
+511 break
FAILS on test_lockf, passes on test_flock -> with open(support.TESTFN,
'w+b') as f:
FAILS on both test_lockf and test_flock (as expected I am guessing) ->
with open(support.TESTFN, 'r+b') as f:
I am not python savvy enough to get it tested using the syntax: ' "with
open('%s', 'wb') as f:" % support.TESTFN, '
+500 with kill_on_error(proc):
+501 "with open('%s', 'wb') as f:" % support.TESTFN,
+502 while True: # synchronize the subprocess
+503 dt = time.monotonic() - start_time
+504 if dt > 60.0:
+505 raise Exception("failed to sync child in
%.1f sec" % dt)
+506 try:
+507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
+508 lock_func(f, fcntl.LOCK_UN)
+509 time.sleep(0.01)
+510 except BlockingIOError:
+511 break
--- run eintr_tester.py ---
File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 502
while True: # synchronize the subprocess
^
IndentationError: unexpected indent
--- eintr_tester.py completed: exit code 1 ---
[-]test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX[/-][+]test_eintr fails on AIX since fcntl functions were modified[/+]vstinner commentedon Jan 7, 2019
Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?
aixtools commentedon Jan 7, 2019
On 07/01/2019 15:46, STINNER Victor wrote:
I was clearly a bit blind.. Now with:
+490 class FNTLEINTRTest(EINTRBaseTest):
+491 def _lock(self, lock_func, lock_name):
+492 self.addCleanup(support.unlink, support.TESTFN)
+493 code = '\n'.join((
+494 "import fcntl, time",
+495 "with open('%s', 'w+b') as f:" % support.TESTFN,
+496 " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
+497 " time.sleep(%s)" % self.sleep_time))
+498 start_time = time.monotonic()
+499 proc = self.subprocess(code)
+500 with kill_on_error(proc):
+501 with open(support.TESTFN, 'w+b') as f:
+502 while True: # synchronize the subprocess
+503 dt = time.monotonic() - start_time
+504 if dt > 60.0:
+505 raise Exception("failed to sync child in
%.1f sec" % dt)
+506 try:
+507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
+508 lock_func(f, fcntl.LOCK_UN)
+509 time.sleep(0.01)
+510 except BlockingIOError:
+511 break
I get - PermissionError - as before.
root@x066:[/data/prj/python/python3-3.8]./python -m test -v test_eintr
== CPython 3.8.0a0 (heads/master-dirty:b4ea8bb080, Jan 1 2019, 18:24:36) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/python3-3.8/build/test_python_3342514
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Run tests sequentially
0:00:00 [1/1] test_eintr
test_all (test.test_eintr.EINTRTests) ...
--- run eintr_tester.py ---
test_flock (main.FNTLEINTRTest) ... ok
test_lockf (main.FNTLEINTRTest) ... ERROR
test_read (main.OSEINTRTest) ... ok
test_wait (main.OSEINTRTest) ... ok
test_wait3 (main.OSEINTRTest) ... ok
test_wait4 (main.OSEINTRTest) ... ok
test_waitpid (main.OSEINTRTest) ... ok
test_write (main.OSEINTRTest) ... ok
test_devpoll (main.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (main.SelectEINTRTest) ... skipped 'need select.epoll'
test_kqueue (main.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (main.SelectEINTRTest) ... ok
test_select (main.SelectEINTRTest) ... ok
test_sigtimedwait (main.SignalEINTRTest) ... ok
test_sigwaitinfo (main.SignalEINTRTest) ... ok
test_accept (main.SocketEINTRTest) ... ok
test_open (main.SocketEINTRTest) ... ok
test_os_open (main.SocketEINTRTest) ... ok
test_recv (main.SocketEINTRTest) ... ok
test_recvmsg (main.SocketEINTRTest) ... ok
test_send (main.SocketEINTRTest) ... ok
test_sendall (main.SocketEINTRTest) ... ok
test_sendmsg (main.SocketEINTRTest) ... ok
test_sleep (main.TimeEINTRTest) ... ok
======================================================================
ERROR: test_lockf (main.FNTLEINTRTest)
----------------------------------------------------------------------
Ran 24 tests in 9.098s
FAILED (errors=1, skipped=3)
--- eintr_tester.py completed: exit code 1 ---
FAIL
======================================================================
FAIL: test_all (test.test_eintr.EINTRTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/data/prj/python/git/python3-3.8/Lib/test/test_eintr.py", line 31, in test_all self.fail("eintr_tester.py failed") AssertionError: eintr_tester.py failedRan 1 test in 9.674s
FAILED (failures=1)
test test_eintr failed
test_eintr failed
== Tests result: FAILURE ==
aixtools commentedon Jan 15, 2019
On 07/01/2019 15:46, STINNER Victor wrote:
vstinner commentedon Jan 15, 2019
I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?
There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.
aixtools commentedon Jan 18, 2019
I’ll make a new PR and delete the news entry.
Sent from my iPhone
aixtools commentedon Jan 21, 2019
Done.
miss-islington commentedon Feb 14, 2019
New changeset b94d4be by Miss Islington (bot) (Michael Felt) in branch 'master':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
b94d4be
Mariatta commentedon Feb 14, 2019
Thanks all!
miss-islington commentedon Feb 14, 2019
New changeset 7e618f3 by Miss Islington (bot) in branch '3.7':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
7e618f3
aixtools commentedon Mar 10, 2019
Could this also be backported to version 3.6?
aixtools commentedon Mar 11, 2019
On 10/03/2019 19:34, Michael Felt wrote:
kadler commentedon May 7, 2021
FYI, the problem here is that AIX fcntl returns EACCES in the case that the lock is held and non-blocking behavior was requested:
https://www.ibm.com/docs/en/aix/7.1?topic=l-lockfx-lockf-flock-lockf64-subroutine
(Note the docs are a bit wonky referring to lockf/lockfx but talking about parameters and fields which apply to fcntl instead)
The lockf/flock APIs provided by AIX handle this appropriately, mapping EACCES to EWOULDBLOCK, but while Python calls the libbsd flock API, it uses its own lockf implementation which calls fcntl directly: https://github.com/python/cpython/blob/main/Modules/fcntlmodule.c#L426