ENH: optimize: add a new method least_squares_lm for curve_fit.

[AtsushiSakai]

Reference issue

Fix #14337

What does this implement/fix?

The current curve_fit has these methods:

1. lm, which is using leastsq
2. trf, which is using least_squares with method='trf'.
3. dogbox, which is using least_squares with method='dogbox'.

So, the current curve_fit is missing a method that is using least_squares with method='lm'.

Both leastsq and least_squares with method='lm' are based on MINPACK’s lmdif and lmder functions,
but as reported in #14337, least_squares with method='lm' has some good optional arguments of least_squares.
(e.g. jac_sparsity for large systems)
So, this PR adds a new method least_squares_lm for curve_fit, which is using least_squares with method='lm' for curve fitting and enhanced unit tests for it.

Additional information

[tupui]

Thanks @AtsushiSakai! Since you had a look at this closely, is there a reason we don't change lm to not use leastsq but directly least_squares? I guess there are backward compatibility concerns, but how much is this a concern? Results are worse, better, API is too different, other?

[AtsushiSakai]

Thank you for your comment @tupui. Yes, I though we should replace leastsq to least_squares with lm option.
But, least_squares is missing some optional arguments which leastsq has: (e.g. full_output, factor, epsfcn, diag, etc.).
We need to add these to least_squares for replacing. Do you think we should try it?

[andyfaff]

But, least_squares is missing some optional arguments which leastsq has: (e.g. full_output, factor, epsfcn, diag, etc.).

It might be worth looking into the feasibility of this. It may prove quite difficult to do this and get all the bugs ironed out. Even if it's possible one trouble is that the least_squares interface then becomes much more complex. For example the least_squares interface might gain an epsfcn keyword, but how would that be handled if diff_step is also specified.
In other words, there's a whole bunch of keywords that are only be active for one method, and they would interact/overlap to some extent with existing keywords.

[tupui]

Maybe instead of trying to fit all the API of one into the API of the other, we could find a minimal set of parameters? i.e. that might mean some deprecation being introduced for the lm option (e.g. if we don't want anymore full_output, calling with full_output would call leastsq and raise a deprecation warning.)
(I am sorry I am not really helpful here as I don't work with these and don't have much sense as of now on how these are used in the wild.)

[andyfaff]

My initial thoughts are that least_squares_lm is the least painful route. But it's a good idea for someone to do that feasibility check

[tupui]

Oh it is for sure. I am just trying, if we can, to avoid a double deprecation loop 😅 Like adding least_squares_lm and later realise we want to remove it in favour of lm.

[dschmitz89]

I might be misunderstanding but from what i understand, all the advanced feature sof least_squares are not available via MINPACK's Levenberg-Marquardt solver. See the docs here:

‘lm’ : Levenberg-Marquardt algorithm as implemented in MINPACK. Doesn’t handle bounds and sparse Jacobians. Usually the most efficient method for small unconstrained problems.

So I am not sure if this PR would only complicate the API.