fixing polynomial and high-Q behavious

[till-schertenleib]

In getx we are fitting over the whole Q-range. I think we want to do the same. One problem I'm running into is that this makes the code choose an rpoly value that smooths S(Q) -->1 over the whole Q range. The code priorities smoothing low-Q ripples (structural signal) over smoothing S(Q) at high Q.
If we don't use a hiQ mast to fit the polynomial, we have to make the high-Q prioritization in some other way.

"Speed" problem: when defining poly_deg = (rpoly * qmax) / pi, poly_deg is almost never an integer. Rounding it to an integer would introduce abrupt changes in the PDF at the half-integer values. We would prefer the PDF to respond smoothly to the rpoly and Qmaxinst parameters. To simulate a polynomial fit at an arbitrary floating-point degree, the correction polynomial is therefore refined twice, for an integer floor and ceiling of nr, and the two fits are then averaged with the weights given by the distance of nr from its integer bounds.

This makes our code much more expensive, because rpoly (and therefore the weighing of the floor and ceiling) changes at each energy. We can't define a single average weighing for all energies, and instead have to fit a polynomial at each energy.

Do we use Qmaxinst or Qmax for defining poly_deg. getx uses the former, so we can do the same. What probably makes sense is to cut the F(Q) at a user-defined Qmax for the FT. --- or we could cut it at a point where F(Q) is close to 0?

Things to double-check by the math geeks:

1. _build_S_from_raw(): we solve coefficients for the polynomial using the weighted matrices that prioritize the high-Q range. We then evaluate the UNWEIGHTED matrix to apply the correction. So we use the coefficients solved on the weighted matrix to then calculate the actual y-values of the polynomial on the unweighted matrix. We have to do that because we don't want the resulting corrected S(Q) to have weird amplified signal at high-Q. Does this make sense?

2. _build_S_from_raw(): we define the y-data to wich the polynomial is fitted as "y = (1.0 - S0[i]) * (fmean_real[i]**2)". We multiply by fmean_real because we want to fit the polynomial to the raw intensities, not SQ. This is how it was done in the original pipeline, but I'm wondering why we aren't just taking the raw intensities directly. This seems to be unnecessary compute here.

3. _reconstruct_Salpha_Snot(): I removed a high-mask anchor in this function. Needs to be double checked. Also, there is a line marked with "!!!!" and it seems like there is some manual override of some sort being done. I have no idea what it's doing or why it's there. But this is in the part that Tieqiong is fixing I think.

4. reconstruct_partials_and_pdfs(): I removed hiQ masks here, and replaced the polydeg with rpoly. There are also some hiQ stats computed (probably for monitoring), and I just replaced the high-Q mask with some fixed number... It shouldn't influence the refinement.

5. RefinementConfig: replaced polydeg with rpoly and removed hiQ mask stuff. Also adjusted Qmin and Qmax. I added a stepsize of 0.05 for rpoly. We can decide if that's reasonable or not.

6. _peak_params(): I added rpoly. I think we just add rpoly to some dictionary here and make sure we can refine it in the next function

7. run_adpdf_with_refinements(): add rpoly stuff and removed mask.

8. in the cell where we actually run the pipline ("v3 runner") we set Qmax=50.0 for some reason. This does not affect the polynomial order, which defines Qmax as the maximum Q value in the Q-array, but it does affect how the Lorch functions in the S_to_Gr functions are defined.

9. We might have to tune the objective functions. I had to change the "smooth_lambda" value in the config file to something lower because otherwise, the code would sometimes just flatten out the whole S(Q) to reach a perfect score. This is something we need to discuss.

[sbillinge]

Yah, this seems pretty messy PR with lots of ipynb. We may want to take a step back and move things to .py to get rid of the notebooks. I am on a flight but I can take a look a bit later