EIGH very very slow --> suggesting an easy fix #9212

This post extends #7339, where @yaroslavvb noticed first that EIGH was surprisingly slow in Scipy vs Numpy.

Anyways first post on Scipy for me!!! Hi all!

I checked Scipy's code on EIGH and how it was implemented. Everything was fine, HOWEVER, you need to add 2 extra lines (literally) of code to make it approx 2-4 times faster than the old Scipy version, and even faster than Numpy's.

Scipy uses:

1. SYEVR if B = None and eigvals = None [Very slow non divide n conquer]
2. SYEVD if B = not None and eigvals = None [Very Fasssttt divide n conquer]
3. SYEVX if B = not None and eigvals = not None. [Slow as well]

There is a slight problem. It should be:

1. SYEVD if B = None (identity) and eigvals = None [Very Fasssttt divide n conquer]
2. SYEVD if B = not None and eigvals = None [Very Fasssttt divide n conquer]
3. SYEVX else

In other words, I propose that Scipy scraps SYEVR completely, and replaces everything with SYEVD. You can keep SYEVR for RAM constraining machines.

An easy fix I tried is:

scipy.linalg.eigh(XTX, b = np.eye(len(XTX), dtype = X.dtype), turbo = True, check_finite = False)

Which performs at most 3-4 times faster than doing this:

scipy.linalg.eigh(XTX, b = None, turbo = True, check_finite = False)

Note, I also tried:

scipy.linalg.eigh(XTX, b = np.eye(len(XTX), dtype = np.int8), turbo = True, check_finite = False)

Which is even slower (meaning you must make identity same dtype as XTX

So, a simple fix in Scipy would be simply:

def eigh(X, b.....):
    if b is None:
        b = np.eye(len(X), dtype = X.dtype)
    ....

In terms of timing,

Also, this post flows on from pytorch/pytorch#11174, where I reported that PyTorch's SVD and GELS was slow and unstable.