Compatibility issues with Numpy 2.4

[Finnjek]

After updating my python environment I noticed that orGUI is not showing correct H K L del gam values under the central plot anymore. Instead of numbers, the displayed value is ERROR for all five entries. I figured that the latest Numpy update to version 2.4 is causing this:

Expired deprecations
Raise TypeError on attempt to convert array with ndim > 0 to scalar
Conversion of an array with ndim > 0 to a scalar was deprecated in NumPy 1.25. Now, attempting to do so raises TypeError. Ensure you extract a single element from your array before performing this operation.

The xyToHKL() function in orGUI.py is used to calculate HKL from the coordinates and seems to yield erroneous values.

    def xyToHKL(x,y):
        #print("xytoHKL:")
        #print("x,y = %s, %s" % (x,y))
        if self.fscan is None:
            return np.array([np.nan,np.nan,np.nan, np.nan, np.nan])
        mu, om = self.getMuOm(self.imageno)
        gamma, delta = self.ubcalc.detectorCal.surfaceAnglesPoint(np.array([y]),np.array([x]), mu)
        #print(self.ubcalc.detectorCal)
        #print(x,y)
        #print(self.ubcalc.detectorCal.tth(np.array([y]),np.array([x])))
        pos = [mu,delta[0],gamma[0],om,self.ubcalc.chi,self.ubcalc.phi]
        pos = HKLVlieg.crystalAngles(pos,self.ubcalc.n)
        hkl = np.concatenate((np.array(self.ubcalc.angles.anglesToHkl(*pos)),np.rad2deg([delta[0],gamma[0]])))
        return hkl
    return xyToHKL

The actual calculation is performed in HKLVlieg.py which is called via self.ubcalc.angles.anglesToHkl(*pos). In case of phi.size=chi.size=omega.size=1, a matrix called PCO is first created and then filled element-wise:

        PCO = np.empty((3, 3), dtype=np.float64)
        PCO[0,0] = coschi*cosomega
        PCO[0,1] = -sinomega*coschi
        PCO[0,2] = -sinchi
        PCO[1,0] = sinchi*sinphi*cosomega + sinomega*cosphi
        PCO[1,1] = -sinchi*sinomega*sinphi + cosomega*cosphi
        PCO[1,2] = sinphi*coschi[0]
        PCO[2,0] = sinchi*cosomega*cosphi - sinomega*sinphi
        PCO[2,1] = -sinchi*sinomega*cosphi - sinphi*cosomega
        PCO[2,2] = coschi*cosphi

This fails with Numpy 2.4 as all sin___ and cos___ objects are arrays and not scalars. Reproduction a MWE of the error with Numpy 2.3 gives the following deprecation warning:

[Finnjek]

The issue can be easily solved by always calling the element [0] of all terms when filling the PCO matrix. However, I am afraid that there may be other issues within HKLVlieg.py regarding Numpy 2.4 compatibility. We should have an eye on that in the future.

[Finnjek]

I found two new issues related to Numpy 2.4 regarding ROI functionality:

• nothing happens when selecting the 'show ROI' dropdown, e.g. you can not make ROIs visible
• integration of static ROIs results in uncaught exception and program crash

I will add a commit with my 'hotfix' solutions, again by selecting explicitly the element [0] of arrays that are (hopefully) supposed to be always one element sized.

[Finnjek]

The function crystalAnglesPoint() was the reason for errors in two cases as it returns the angles gamma_cry, delta and alpha_i_cry in one-element sized arrays. Instead of fixing this in the orGUI.py code, we should consider doing some changes in DetectorCalibration.py as this is where the issue originates. Is it correct that the returned angles are always intended to give only one value, or is the function also intended to calculate the angles for multiple positions and return arrays longer than 1?

[tifuchs]

Yes, crystalAnglesPoint() takes either single points or arrays.

Great work on fixing these bugs! I will work on this soon, too. Right now I am occupied with more beamtime...

[tifuchs]

calling flatten() on an 0-d array will make it an 1-d array.
in angles.anglesToHkl:

        sinphi = np.sin(phi.flatten()); cosphi = np.cos(phi.flatten())
        sinchi = np.sin(chi.flatten()); coschi = np.cos(chi.flatten())
        sinomega = np.sin(omega.flatten()); cosomega = np.cos(omega.flatten())

Instead of flatten all arrays without checking for size, we should explicitly check a.size > 1: a.flatten(), else not.

[tifuchs]

I added ruff (pip install ruff pyupgrade) linting rules to the repo in bb89a41.

running ruff check . --fix should find and refactor easy syntax changes for newer python and numpy versions (for example the np.float -> float changes could have been solved automatically with this tool). It will also emit syntax 'errors'. Currently: still ~1000 remaining that cannot be changed easily.

Current python target version is 3.10 / numpy 2.x for release 1.5. (i.e. python <= 3.9 will be dropped)

[tifuchs]

    def anglesToHkl(self,alpha,delta,gamma,omega,chi,phi):
        alpha = np.asarray(alpha)
        delta = np.asarray(delta)
        gamma = np.asarray(gamma)
        omega = np.asarray(omega)
        chi = np.asarray(chi)
        phi = np.asarray(phi)
        ang = [alpha,delta,gamma,omega,chi,phi]

        shape = max([a.shape for a in ang])

        #[_, _, _, OMEGA, CHI, PHI] = createVliegMatrices([None,None,None,omega,chi,phi])
        #hkl = np.empty((*shape,3))
        K = self._ubCalculator.getK()
        UBi = np.linalg.inv(self._ubCalculator.getUB())

        Qalp = self.QAlpha(alpha,delta,gamma)
        Qalp = np.ascontiguousarray(Qalp.reshape((-1,3)))

        #now UBi * PHIi * CHIi * OMEGAi * Q_alpha
        if omega.ndim > 0:
            omega = omega.ravel()
        if chi.ndim > 0:
            chi = chi.ravel()
        if phi.ndim > 0:
            phi = phi.ravel()
        #calculate PHIi @ CHIi @ OMEGAi
        sinphi = np.sin(phi); cosphi = np.cos(phi)
        sinchi = np.sin(chi); coschi = np.cos(chi)
        sinomega = np.sin(omega); cosomega = np.cos(omega)

        if phi.size > 1 or chi.size > 1 or omega.size > 1:
            # this will be expensive, could optimize here in the future
            PCO = np.empty((np.prod(shape), 3, 3), dtype=np.float64)
            PCO[:,0,0] = coschi*cosomega
            PCO[:,0,1] = -sinomega*coschi
            PCO[:,0,2] = -sinchi
            PCO[:,1,0] = sinchi*sinphi*cosomega + sinomega*cosphi
            PCO[:,1,1] = -sinchi*sinomega*sinphi + cosomega*cosphi
            PCO[:,1,2] = sinphi*coschi
            PCO[:,2,0] = sinchi*cosomega*cosphi - sinomega*sinphi
            PCO[:,2,1] = -sinchi*sinomega*cosphi - sinphi*cosomega
            PCO[:,2,2] = coschi*cosphi
        else:
            PCO = np.empty((3, 3), dtype=np.float64)
            PCO[0,0] = coschi*cosomega
            PCO[0,1] = -sinomega*coschi
            PCO[0,2] = -sinchi
            PCO[1,0] = sinchi*sinphi*cosomega + sinomega*cosphi
            PCO[1,1] = -sinchi*sinomega*sinphi + cosomega*cosphi
            PCO[1,2] = sinphi*coschi
            PCO[2,0] = sinchi*cosomega*cosphi - sinomega*sinphi
            PCO[2,1] = -sinchi*sinomega*cosphi - sinphi*cosomega
            PCO[2,2] = coschi*cosphi

        #calculate UBi * PHIi * CHIi * OMEGAi * Q_alpha
        hkl = np.einsum('...ij,...j->...i', (UBi @ PCO), Qalp)
        #hkl = np.matmul(UBi,np.matmul(PHIi,np.matmul(CHIi,np.matmul(OMEGAi,DEL_GAM_minALP.T)))).T.reshape((*shape,3))
        hkl = hkl.reshape((*shape,3))
        return hkl[...,0], hkl[...,1], hkl[...,2] # h k l

[tifuchs]

Nice catch with the np.diff on 1d array with size = 2 in
roi_hsize1_a[i] = int(np.abs(np.diff(croi_key[0])[0]))