You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BUG: float64 nanmin() returns NaN on little-endian MIPS (mipsel, mips64el) #23158
import sys, numpy; print(numpy.version); print(sys.version)
1.24.1
3.11.1 (main, Dec 31 2022, 10:23:59) [GCC 12.2.0]
print(numpy.show_runtime())
WARNING: threadpoolctl not found in system! Install it by pip install threadpoolctl. Once installed, try np.show_runtime again for more detailed build information
[{'simd_extensions': {'baseline': [], 'found': [], 'not_found': []}}]
None
Context for the issue:
This is a fundamental bug in a data type, breaking higher level verification tests in other packages.
Is there a GCC compile farm machine with the appropriate configuration to debug this? I checked https://cfarm.tetaneutral.net/machines/list/ and tried host gcc230, which also runs Debian, but it is Big Endian it seems:
Architecture: mips64
Byte Order: Big Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
BogoMIPS: 2000.00
L1d cache: 32K
L1i cache: 78K
L2 cache: 512K
NUMA node0 CPU(s): 0-3
I'm not aware of any public machines, I'm afraid. I can test things on Debian's mips porterbox, if that helps. I haven't tried to debug this, at all, beyond pinpointing the issue in numpy.
I believe that QEMU's support for mips64el and mipsel are both usable.
I tried to have a bit of a look, but need to try to get emulation running. It does seem like this this should just end up using fmin in C, unless it goes down some interesting SIMD paths (but I don't think it does from np.show_runtime()). It would be surprising to fail though.
Whenever I return to this, I don't really see how this can happen (sorry I didn't get to a working qemu/native setup, yet).
Do you have the NumPy build setup/flags ready or are they the defaults? Maybe there is something simple like someone being a bit overzealous on optimization and enabling -ffast-math?!
(In godbold, -ffast-math seems to replace the fmin call with __ledf2 which is not NaN aware, there may be other ways to trigger that I guess. EDIT: I doubt it, but am not sure some of the following instructions doesn't do something about it.)
Thanks. Looking for instance at the sucessful ppc64el build log, I see gcc12 but not the specific flags used. With that, I do not see any flag in the mips64el build that should cause problems.
Maybe disabling all kinds of optimization including disabling optimization level 3 can be used as a workaround for this issue through the following command:
$ objdump /home/stefanor/numpy-1.24.2/build/lib.linux-mips64-cpython-311/numpy/core/_multiarray_umath.cpython-311-mips64el-linux-gnuabi64.so -t | grep fmin
00000000002f61b8 l O .data 00000000000000a8 fmin_functions
000000000031bc50 l O .bss 00000000000000a8 fmin_data
00000000002f4258 l O .data 000000000000003f fmin_signatures
00000000001a30a0 l F .text 00000000000000dc LONGDOUBLE_fmin
00000000001a29c0 l F .text 00000000000000d0 FLOAT_fmin
0000000000199e58 l F .text 00000000000000e8 CFLOAT_fmin
00000000001a2c90 l F .text 00000000000000d0 DOUBLE_fmin
000000000019d8b8 l F .text 00000000000001dc CLONGDOUBLE_fmin
000000000019af28 l F .text 00000000000000e8 CDOUBLE_fmin
0000000000197fc8 l F .text 00000000000000f8 HALF_fmin
00000000001917d0 l F .text 0000000000000018 TIMEDELTA_fmin
00000000001912c8 l F .text 0000000000000084 DATETIME_fmin
0000000000280d70 F *UND* 0000000000000000 fminl@GLIBC_2.2
0000000000281a80 F *UND* 0000000000000000 fmin@GLIBC_2.2
0000000000281db0 F *UND* 0000000000000000 fminf@GLIBC_2.2
Honestly, I see nothing weird. I still suspect it is a compiler issue somehow. But I have no idea, I guess it jumps to t9,-23064(gp) which should be the call to s1 = fmin(s2, s3) (s1, s2, s3 being the registers)? But is that a jump into an fmin version that is included into NumPy rather than external?
We could put a work-around in place if necessary, but right now I don't even know how that would look like, since the looks perfectly fine.
I don't really see why/how, but considering that there is nothing obvious here, maybe? If float("nan") representation has to do with it, that might at least explain why NumPy it sounds like the NumPy test-suite apparently doesn't see this (much).
I.e. maybe code using np.nan works, or at least doing np.fmin.reduce(np.array([0.0, 1.0]) / np.array([0., 2.])) works?!
So @wzssyqa is right and MIPS is weird, which supposedly can be specified what NaNs you want, but no idea what helps but maybe you should just compile everything with -mnan=2008.
There are layers of problems here I think:
NumPy uses the "right" legacy NaN. This looks like an accident to me, because I think it is generated by casting a wrong single precision NaN.
Python uses __builtin_nan(""), that can probably be tweaked.
Using NAN C99 probably would work for everyone nowadays, but Python may want to support systems that don't have quiet NaNs, so probably would need to use an #ifdef NAN.
I suspect the fmin is technically also wrong for not ignoring the signalling NaN as well, although signalling NaNs seem pretty useless to me and we do not really bother with them anyway.
Do you/we care enough to push fixes into Python for this? Maybe we should just add that MIPS -mnan=2008 config if that is pragmatic. If Python uses it, NumPy would presumably inherit it.
Or, you fix the definition of NAN to use C99 NAN when available (for NumPy I would be happy to just use it always until someone dares to complain, but Python may have different ideas for supporting strange platforms; I find it hard to believe that a platform would have no quiet NaNs but implement signaling ones...).
Python uses __builtin_nan(""), that can probably be tweaked.
Interestingly, using that instead of NAN doesn't seem to trigger the fmin bug (in a trivial C example). So there's something deeper going on, there.
But yeah, I think I'm happy to call this the MIPS port's problem, and let them drive a solution. I can't even compile with -mnan=2008 on current Debian unstable...
I may have misread the Python code. In the dark dark depths, there is this function: _Py_dg_stdnan. Which is used if and only if _PY_SHORT_FLOAT_REPR == 1. I have no clue what that means! But that function would return the wrong NaN (fmin doing the right thing or not, its not a quiet one MIPS).
This code was recently touched here: python/cpython#31171 but neither before nor after can I figure out if/why that path might actually be taken on MIPS (I guess due to a configure mistake?). I suppose a python -m sysconfig | grep DOUBLE might shed light on whether it is taken.
But the thing about all of this is. Below the nice neat comment (one place where it is used):
/* Constant nan value, generated in the same way as float('nan'). */
/* We don't currently assume that Py_NAN is defined everywhere. */
is an unguarded function that also uses Py_NAN. So, maybe do have a look at the sysconfig, and then maybe we should just delete that function from cPython because deleting 100 lines of dead code is always nice :).
I misunderstood the _PY_SHORT_FLOAT_REPR var and that path is always taken. So it definitely generates the wrong NaN.
The minimal Python fix would be to just modify _Py_dg_stdnan to return Py_NAN I think. I suspect Py_NAN is guaranteed to be defined and this is all just a huge mess that was never cleaned up in Python, mixing up proper IEEE floats and the definition of NaN. Maybe, that is fine for whether math.nan is exposed, but for the rest it seems all unnecessary complexity.
Closing this issue, it clearly isn't NumPy here. Its MIPS, and Python, and the compiler. And potentially all three working together with each doing something wrong.
Activity
tylerjereddy commentedon Feb 4, 2023
Is there a GCC compile farm machine with the appropriate configuration to debug this? I checked https://cfarm.tetaneutral.net/machines/list/ and tried host
gcc230, which also runs Debian, but it is Big Endian it seems:stefanor commentedon Feb 5, 2023
I'm not aware of any public machines, I'm afraid. I can test things on Debian's mips porterbox, if that helps. I haven't tried to debug this, at all, beyond pinpointing the issue in numpy.
I believe that QEMU's support for mips64el and mipsel are both usable.
seberg commentedon Feb 15, 2023
I tried to have a bit of a look, but need to try to get emulation running. It does seem like this this should just end up using
fminin C, unless it goes down some interesting SIMD paths (but I don't think it does fromnp.show_runtime()). It would be surprising to fail though.seberg commentedon Apr 12, 2023
@stefanor any chance you could make sure that
fminitself isn't broken? My guess is, something like this should do:Compile with
gcc -lmand run. I really don't see how we will not end up withfmin.stefanor commentedon Apr 17, 2023
On both mips64el and mipsel:
seberg commentedon Apr 21, 2023
Whenever I return to this, I don't really see how this can happen (sorry I didn't get to a working qemu/native setup, yet).
Do you have the NumPy build setup/flags ready or are they the defaults? Maybe there is something simple like someone being a bit overzealous on optimization and enabling
-ffast-math?!(
In godbold,EDIT: I doubt it, but am not sure some of the following instructions doesn't do something about it.)-ffast-mathseems to replace thefmincall with__ledf2which is not NaN aware, there may be other ways to trigger that I guess.stefanor commentedon Apr 21, 2023
Looks like it's all defaults.
You can see them in a build log: https://buildd.debian.org/status/fetch.php?pkg=numpy&arch=mips64el&ver=1%3A1.24.2-1&stamp=1675927314&raw=0
mattip commentedon Apr 25, 2023
I see gcc12 and these flags:
Do we have any known successful little-endian systems?
stefanor commentedon Apr 26, 2023
We don't see the issue on any other Debian platforms: https://buildd.debian.org/status/logs.php?pkg=silx&ver=1.1.0%2Bdfsg-4 (there is a test failure on s390x, but that's unrelated)
mattip commentedon Apr 26, 2023
Thanks. Looking for instance at the sucessful ppc64el build log, I see gcc12 but not the specific flags used. With that, I do not see any flag in the mips64el build that should cause problems.
seiko2plus commentedon Apr 26, 2023
Maybe disabling all kinds of optimization including disabling optimization level 3 can be used as a workaround for this issue through the following command:
through pip:
pip install --no-use-pep517 --global-option=build \ --global-option="--disable-optimization" ./This issue may be related to the following manual unroll which can be disabled by build option
--disable-optimization:numpy/numpy/core/src/umath/loops_minmax.dispatch.c.src
Lines 368 to 436 in bc8cf11
2 remaining items
seberg commentedon Apr 27, 2023
I am not sure that it makes sense to try to spend much time on this TBH, but just to see... one more question. Given:
as
<file_path>, what does:give you? On x86 (without optimizatoin) it gives for example:
Which calls
fminin there, which we suspected is good from the C program (but who knows...).(
objdump <file> -t | grep fminshould show that there is only theDOUBLE_fminversion around)stefanor commentedon Apr 27, 2023
FWIW, a bisect pointed to #20131 as the origin point of this bug (which may not be that surprising, it was a massive rewrite of the relevant code).
The disassemble (from 1.24.2):
And yes, only one
DOUBLE_fmin:seberg commentedon Apr 28, 2023
Honestly, I see nothing weird. I still suspect it is a compiler issue somehow. But I have no idea, I guess it jumps to
t9,-23064(gp)which should be the call tos1 = fmin(s2, s3)(s1, s2, s3 being the registers)? But is that a jump into anfminversion that is included into NumPy rather than external?We could put a work-around in place if necessary, but right now I don't even know how that would look like, since the looks perfectly fine.
wzssyqa commentedon Apr 30, 2023
Is it about the encoding of nan?
If so, I guess it is due to the NaN-legacy vs NaN2008 problem?
For NaN-legacy hardware, the encoding of qNaN is different with other hardwares.
I will try to dig it out.
seberg commentedon May 2, 2023
I don't really see why/how, but considering that there is nothing obvious here, maybe? If
float("nan")representation has to do with it, that might at least explain why NumPy it sounds like the NumPy test-suite apparently doesn't see this (much).I.e. maybe code using
np.nanworks, or at least doingnp.fmin.reduce(np.array([0.0, 1.0]) / np.array([0., 2.]))works?!stefanor commentedon May 4, 2023
It seems to work, and yes it's different:
seberg commentedon May 5, 2023
Fine, so I guess you get: https://sourceware.org/binutils/docs/as/MIPS-NaN-Encodings.html
So @wzssyqa is right and MIPS is weird, which supposedly can be specified what NaNs you want, but no idea what helps but maybe you should just compile everything with
-mnan=2008.There are layers of problems here I think:
__builtin_nan(""), that can probably be tweaked.NANC99 probably would work for everyone nowadays, but Python may want to support systems that don't have quiet NaNs, so probably would need to use an#ifdef NAN.fminis technically also wrong for not ignoring the signalling NaN as well, although signalling NaNs seem pretty useless to me and we do not really bother with them anyway.Do you/we care enough to push fixes into Python for this? Maybe we should just add that MIPS
-mnan=2008config if that is pragmatic. If Python uses it, NumPy would presumably inherit it.Or, you fix the definition of NAN to use C99
NANwhen available (for NumPy I would be happy to just use it always until someone dares to complain, but Python may have different ideas for supporting strange platforms; I find it hard to believe that a platform would have no quiet NaNs but implement signaling ones...).stefanor commentedon May 5, 2023
Interestingly, using that instead of
NANdoesn't seem to trigger thefminbug (in a trivial C example). So there's something deeper going on, there.But yeah, I think I'm happy to call this the MIPS port's problem, and let them drive a solution. I can't even compile with
-mnan=2008on current Debian unstable...Thanks for your time :(
seberg commentedon May 5, 2023
I may have misread the Python code. In the dark dark depths, there is this function: _Py_dg_stdnan. Which is used if and only if
_PY_SHORT_FLOAT_REPR == 1. I have no clue what that means! But that function would return the wrong NaN (fmin doing the right thing or not, its not a quiet one MIPS).This code was recently touched here: python/cpython#31171 but neither before nor after can I figure out if/why that path might actually be taken on MIPS (I guess due to a configure mistake?). I suppose a
python -m sysconfig | grep DOUBLEmight shed light on whether it is taken.But the thing about all of this is. Below the nice neat comment (one place where it is used):
is an unguarded function that also uses Py_NAN. So, maybe do have a look at the sysconfig, and then maybe we should just delete that function from cPython because deleting 100 lines of dead code is always nice :).
seberg commentedon May 5, 2023
I misunderstood the
_PY_SHORT_FLOAT_REPRvar and that path is always taken. So it definitely generates the wrong NaN.The minimal Python fix would be to just modify
_Py_dg_stdnanto returnPy_NANI think. I suspectPy_NANis guaranteed to be defined and this is all just a huge mess that was never cleaned up in Python, mixing up proper IEEE floats and the definition of NaN. Maybe, that is fine for whethermath.nanis exposed, but for the rest it seems all unnecessary complexity.seberg commentedon May 5, 2023
Closing this issue, it clearly isn't NumPy here. Its MIPS, and Python, and the compiler. And potentially all three working together with each doing something wrong.
MAINT: Rely on Py_NAN and Py_HUGE_VAL being correctly defined
skip test_axis_nan_policy_axis_is_None if RuntimeWarning emitted
TST: `test_axis_nan_policy`: skip on unexpected exceptions (#23264)