Avoid specialized assert formatting when we detect that __eq__ is overridden

The issue reported for introspecting numpy arrays #5347, is still present for dataclasses when invoking assertrepr_compare.

When testing:

from dataclasses import dataclass
import numpy as np


@dataclass
class A:
    a: np.ndarray
    
    def __eq__(self, other):
        return np.array_equal(self.a, other.a)


def test_fails():
    assert A(np.array([1,2])) == A(np.array([2,1]))

Instead of an "Drill down" explanation you get:

   def test_fails():
>       assert A(np.array([1,2])) == A(np.array([2,1]))
E       assert A(a=array([1, 2])) == A(a=array([2, 1]))
E         (pytest_assertion plugin: representation of details failed: .../_pytest/assertion/util.py:430: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
E          Probably an object has a faulty __repr__.)

(Ubuntu 21.10, pytest 6.25, python 3.9)

One way to solve it: since pytest only sees difference on repr perhaps instead of comparing arr1 == arr2 when determining if the fields are different, perhaps it should consider whether repr(arr1) == repr(arr2) which sidesteps the issue with array not returning bool for comparison.

Instead of

pytest/src/_pytest/assertion/util.py

Lines 444 to 448 in 14b79a6

    
           for field in fields_to_check: 
        
               if getattr(left, field) == getattr(right, field): 
        
                   same.append(field) 
        
               else: 
        
                   diff.append(field)

one could do

    for field in fields_to_check:
        if repr(getattr(left, field)) == repr(getattr(right, field)):
            same.append(field)
        else:
            diff.append(field)

If the __repr__s are the same and the object are not the same (or vice versa), then the explanation shown by pytest will regardless be misleading.

probably the more correct solution here is to avoid pytest's specialized assert formatting when it detects that __eq__ is overridden (since it's not going to accurately represent the underlying equality comparins)

it might also not be a bad idea for pytest to check if its numpy and use the numpy equals there as we do check numpy in a number of places - im under the impression a lot of the numpy handling should get a own plugin module by now

@RonnyPfannschmidt

@RonnyPfannschmidt that would create a bug in the other direction in the default case (since dataclass is just going to use == and if we used numpy's comparison then it would be different)

probably the more correct solution here is to avoid pytest's specialized assert formatting when it detects that __eq__ is overridden (since it's not going to accurately represent the underlying equality comparins)

This seems like the most robust solution.

@RonnyPfannschmidt

@RonnyPfannschmidt I can try to work on this one if anyone haven't yet

Any tip on how we can determine if the __eq__ method is overridden?
I've tried using __code__ attribute but I don't think this is the right direction

@hynek

off hand, i'm not sure, - @hynek from attrs may have a hint

it seems to be a little tricky since the method is generated -- one way would maybe be to compare the co_filename of __eq__.__code__ (which appears to be <string> when it's overridden, or whatever the module filename is otherwise)

@willmcgugan

I remember that @willmcgugan did similar work for rich about detecting if repr was overwritten, so maybe he can help.

@hynek

@hynek Thanks!
I think I found it: https://github.com/willmcgugan/rich/blob/07d51ffc1aee6f16bd2e5a25b4e82850fb9ed778/rich/pretty.py#L68

When I look deeper into the problem, I see it comes in many forms, consider the following code:

import numpy as np
import attr

@attr.s
class SimpleDataObject:
    a = attr.ib(eq=attr.cmp_using(eq=np.array_equal))


left = SimpleDataObject(np.array([1, 2]))
right = SimpleDataObject(np.array([2, 1]))
assert right == left

Running this code under a test will also produce the same problem mentioned in the issue:

        left = SimpleDataObject(np.array([1, 2]))
        right = SimpleDataObject(np.array([2, 1]))
>       assert right == left
E       assert SimpleDataObject(a=array([2, 1])) == SimpleDataObject(a=array([1, 2]))
E         (pytest_assertion plugin: representation of details failed: c:\users\yuvals\pytest\src\_pytest\assertion\util.py:457: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
E          Probably an object has a faulty __repr__.)

Apart from checking if the dataclass or the attrs.define decorated class (attr.s seems to ignore the customdefined __eq__) has a custom __eq__ method, should we also consider handling attributes which raise an exception during there comparison, something like:

>       assert right == left
E       AssertionError: assert SimpleDataObject(a=array([2, 1])) == SimpleDataObject(a=array([1, 2]))
E         Attributes which raised an exception:
E         ['a']