Nearly equal with relative and absolute tolerance #3152

dvd101x · 2024-02-10T04:39:42Z

Hi,

This initial commit is intended for exploration of #2838. It overwrites the current nearlyEqual completely to be as similar to python's Math.isclose as possible (according to the description in the documentation)

It currently fails 6 tests:

2 tests in rotationMatrix
1 test in nearlyEqual (for bigNumber)
3 tests in number / nearlyEqual

No effort was made to either adapt the tests or fix the function. Everything is left as is to review the failing tests and aid in the decision making.

Please note that the functions are changed abruptly only for exploration and can be returned to their original form upon review.

Also no calls for nearlyEqual were modified, thus only relative tolerance is being used. If the default values are changed to relTol = Number.EPSILON, absTol = 1e-12 then a few more tests fail.

josdejong · 2024-02-14T10:57:48Z

Thanks, this is a great start!

Some thoughts:

Only 8 failing unit tests is not that much :). I don't see failing tests related to rotationMatrix though, I see 4 tests for eigs failing?
I expect more tests to fail as soon as we update all places where nearlyEqual(...) is invoked to pass both relTol and absTol.
I think we may need to add some additional unit tests to test relTol and absTol individually.
We should decide upon good default values relTol and absTol, and how we map values for both when a user configures epsilon (for backward compatibility). Would it make sense to default to relTol = epsilon ?? 1e-12 and absTol = epsilon ?? 1e-12?
In general I like to use full words rather than abbreviations, like relativeTolerance and absoluteTolerance. However in this case the words are quite long. Still thinking about that.
I see the tests where run on node.js 16, I don't understand why, it should only run node 18 and 20 (see config)? Is something in this branch not up to date?

dvd101x · 2024-02-18T01:06:39Z

That's odd, I don't see eigs failing, maybe this is not up to date.
I agree
I agree
I think the current behavior is similar to having an absTol about 1000 times smaller than relTol. I see on the julia docs for isaprox some ways to calculate those default values.
I agree these names don't match mathjs very well just took the python names and got a warning related to the camel case. I'm sure there are more appropriate names. Regarding abbreviations it could be argued that config is an abbreviation.
At some point I saw that line trying to update in my PR but I returned it to what it was. Probably it was my mistake.

dvd101x · 2024-02-18T01:54:37Z

A quick comparison for names and values.

language	function	relative tolerance	absolute tolerance
python	isclose	rel_tol = 1e-09	abs_tol = 0.0
numpy	isclose	rtol = 1e-05	atol = 1e-08
julia	isapprox	rtol = atol>0 ? 0 : √eps	atol = 0

josdejong · 2024-02-21T12:11:40Z

(1) You can see the test results by clicking them in the list with checks in this PR, for example https://github.com/josdejong/mathjs/actions/runs/7853061106/job/21432241083?pr=3152. I guess indeed that you somehow have an old version or so (that may also explain why the PR is still testing on Node 16).

(5) ha ha, that's true. I'm ok with either absTol / relTol or absoluteTolerance / relativeTolerance, I don't have a very strong opinion in this regard (in general, I like the non abbreviated name, but in this case it is quite long).

dvd101x · 2024-02-26T22:53:05Z

(1) I was at v 11.9.1 (sorry about that). Should be updated to v12.4.0 by now.
(5) Ok

Now that it's updated, I see the following errors:

round as predicted at #3136
eigs
round

What would be the next steps?

josdejong · 2024-02-29T15:16:53Z

I think the next step is to work out this PR in detail: update all places where nearlyEqual(...) is invoked, and then fix all failing unit tests.

dvd101x · 2024-02-29T17:31:18Z

Ok, thanks! I will do that and report back.

dvd101x · 2024-04-05T04:37:36Z

Hi, just reporting back.

Progress has been slower than what I originally expected but I'm still on it.

josdejong · 2024-04-05T07:30:53Z

Thanks for the update David, good to know.

dvd101x · 2024-04-06T05:12:49Z

Hi Jos

I have some good news!

I found a good fit with a relative tolerance the size of the current config.epsilon 1e-12 and an absolute tolerance of 1e-15. I did some empirical tests and works similarly in most cases.

In the latest submit I made all calls for nearlyEqual to use relTol = config.epsilon and absTol = config.epsilon * 1e-3 and this eliminates all the failing tests except for the nearlyEqual tests.

I would like your advise with the following:

Shall I remove config.epsilon in favor of config.relTol and config.absTol? (naming to be defined)
Shall I change the failing nearlyEqual tests to accommodate for the proposed nearlyEqual or make a compromise between the proposed nearlyEqual and the current tests?
Shall I make a mathjs function nearlyEqual available so that the user can choose absTol and relTol without changing config or shall it be a different topic outside this PR?

Appendix

Here is the mathjs code I used to test for the tolerances: (this works on the current version 12.4.1

config({epsilon:1e-12});

# Define function nearly equal
nearlyEqual(a, b, relTol, absTol) = abs(a-b) <= max(relTol * max(abs(a), abs(b)), absTol);

# Define rtol and atol 
rtol = config().epsilon;
atol = rtol * 1e-3;

# Tested various values of x
X = [0, 0.1, 0.5, 1, 10, 1e3, 1e6, 1e12];
x = X[1];

# Ys
powers = -2:-1:-16;
Y = x + map(powers, _(d) = pow(10, d));

# Proposed
nearlyEqualResults = map(Y, _(y) = nearlyEqual(x, y, rtol, atol));

# Current
equalResults = x == Y;

# These are the powers where the current nearly equals differ from the proposed one
differences = nearlyEqualResults != equalResults;

result = {
  powersFailing:powers[differences],
  x:x,
  current:equalResults[differences],
  proposed:nearlyEqualResults[differences],
  rtol:rtol,
  atol:atol
};

print("Testing with:\n
rtol: $rtol
atol: $atol
x: $x\n
yields:
powers faling: $powersFailing
proposal: $proposed
current: $current
", result)

I read the following document to realize that nearlyEqual might need to be one of the mathjs functions.

MDN: Number.EPSILON

josdejong · 2024-04-06T09:34:26Z

Yay, that sounds promising!

Yes, though we do need backward compatibility in the function config(). When people are still using .epsilon, we can translate it internally into the new options relTol and abstol and output a console warning about the deprecation.
I think we have to adjust the failing nearlyEqual tests. This will be a breaking change, and I suppose the failing tests are due to the behavior now being different, so the tests should be adapted accordingly. (please correct me if I'm wrong)
I'm not sure if we should expose the nearlyEqual function, it may make sense 🤔. I prefer to discuss that in a separate topic to keep this PR focused, OK?

Looking at the code example, you could definitely use #2953 🤣

dvd101x · 2024-04-06T15:17:45Z

Thanks!

OK: sounds good, I was worried it was too much change.
OK: most are due to the new formula. There is one that might be debatable that says nearlyEqual: should do exact comparison when epsilon is null or undefined. In my opinion it should resort to default values of relTol and absTol when they are null or undefined
OK: Will make a separate discussion.

Absolutely! I was trying to change my point of view about the shorthand and I did.

PD: Also could have used #2701 to cycle through the values of X

josdejong · 2024-04-10T14:01:58Z

👍

I see you're pushing new commits. Please let me know when the PR is ready for review (or if you hit new challenges).

dvd101x · 2024-04-11T17:30:39Z

Hi Jos,

Thanks for the follow up.

I'm still tweaking some stuff and will let you know when it's ready for review.

dvd101x · 2024-04-12T03:47:45Z

Hi Jos, this is ready for review

josdejong · 2024-04-18T17:30:50Z

Thanks, that is good news! I'll look into it asap.

josdejong

This looks really solid David, thanks!

Hardly any existing unit test breaks with the new behavior, I'm kind of (happily) surprised by that. I had expected some changes with edge cases of the rounding functions. Or am I over looking something?

I made a couple of inline comments, can you have a look at those?

src/core/function/config.js

josdejong · 2024-04-24T15:54:40Z

types/index.d.ts

@@ -4296,7 +4298,8 @@ export interface Help {
 }

 export interface ConfigOptions {
-  epsilon?: number
+  relTol?: number
+  absTol?: number


I think we need to keep epsilon here and mark it as deprecated, else existing TypeScript code that uses epsilon will break (and the backward compatibility doesn't work).

Thanks for catching this as I'm not yet familiar with typescript

Ok, just did.

How do I mark it as deprecated?

Like this:

/** * @deprecated Use `relTol` and `absTol` instead */ epsilon?: number

Thanks!
It's done now.

josdejong · 2024-04-24T15:55:49Z

test/unit-tests/utils/number.test.js

+
+      assert.strictEqual(nearlyEqual(1.2 + 1e-7, 1.2), false)
+      assert.strictEqual(nearlyEqual(1.2 + 1e-7, 1.2, undefined), false)
+      assert.strictEqual(nearlyEqual(1.2 + 1e-7, 1.2, undefined, undefined), false)


Can you add a new unit test that validates the backward compatibility, using the deprecated epsilon?

Yes of course,

I just included them on config.test.js but found an issue with console.warn that I don't know how to test and also breaks the pattern of the tests that now looks like this.

.....................................Warning: The configuration option "epsilon" is deprecated. Use "relTol" and "absTol" instead. .....................,,,,,....................... ......................................................................................

This is my error but I don't know how to fix it.

Ah, yes. We can do two things:

Integrate and use some spy library for mocha, like Sinon

Temporarily replacing console.warn with our own custom function, and after the test restore the original console.warn

Option 1 would be best, but only if it is simple to setup, otherwise option 2 is fine with me. Option 2 is a bit ugly: when the test throws an exception the original console.warn is not restored, therefore it's not my first choice.

Thanks!

I will research sinon and report back.

Hi, it seems to be working now, please review.

dvd101x · 2024-04-25T16:48:23Z

Hi Jos, thanks for the thorough review!

I'm also glad there where not so many errors. The reason might be that the values for the new tolerances were picked to be close to the existing behavior. Maybe at some point in the future the values might change and there will be more discrepancies but at least there is the option to adjust as needed case by case.

I will work on the inline comments and report back probably during the weekend.

dvd101x · 2024-04-27T05:29:34Z

Hi Jos, it's ready for review again. I included some inline comments with a few questions.

josdejong

Thanks for the updates! I've replied to the two left over comments, can you have a look at them again?

dvd101x · 2024-05-12T16:39:06Z

Hi Jos, the two left comments were addressed.

Please review.

dvd101x added 6 commits February 9, 2024 21:09

nearlyEqual with absolute and relative tolerances

1b7547e

Format

4dfea1d

nearlyEqual for bigNumber

a254cee

Added skip for NaN

704ba5c

Reduce diff a bit

706154b

Issue with examples in jsdcos

4ba0e8c

josdejong added this to the v13 milestone Feb 14, 2024

josdejong mentioned this pull request Feb 14, 2024

v13 breaking changes #2800

Open

8 tasks

BrianFugate mentioned this pull request Feb 22, 2024

Fixes #3100 Bug on simple rounding #3136

Merged

Merge branch 'develop' into nearlyEqual2

71cfe28

Updated all calls for nearlyEqual

a761260

dvd101x added 3 commits April 6, 2024 10:21

Fixed failing tests

a073665

Changed epsilon to relTol, absTol

05aab2b

Changed references to epsilon in docs and tests

e7eb46e

Added warning for config.epsilon

8fc4c4d

Merge branch 'develop' into nearlyEqual2

d848310

Merge branch 'develop' into nearlyEqual2

a3e2925

josdejong requested changes Apr 24, 2024

View reviewed changes

dvd101x added 2 commits April 26, 2024 20:58

Fix warning in zeta.test

a4ac4a8

Added config test

b164642

josdejong self-requested a review April 29, 2024 08:54

josdejong reviewed Apr 29, 2024

View reviewed changes

Added sinon to test console.warn

8ac2548

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nearly equal with relative and absolute tolerance #3152

Nearly equal with relative and absolute tolerance #3152

dvd101x commented Feb 10, 2024 •

edited

josdejong commented Feb 14, 2024

dvd101x commented Feb 18, 2024 •

edited

dvd101x commented Feb 18, 2024

josdejong commented Feb 21, 2024

dvd101x commented Feb 26, 2024

josdejong commented Feb 29, 2024

dvd101x commented Feb 29, 2024

dvd101x commented Apr 5, 2024

josdejong commented Apr 5, 2024

dvd101x commented Apr 6, 2024

josdejong commented Apr 6, 2024

dvd101x commented Apr 6, 2024

josdejong commented Apr 10, 2024

dvd101x commented Apr 11, 2024

dvd101x commented Apr 12, 2024

josdejong commented Apr 18, 2024

josdejong left a comment

josdejong Apr 24, 2024

dvd101x Apr 27, 2024 •

edited

josdejong Apr 29, 2024

dvd101x May 12, 2024

josdejong Apr 24, 2024

dvd101x Apr 27, 2024 •

edited

josdejong Apr 29, 2024

dvd101x May 4, 2024

dvd101x May 12, 2024

dvd101x commented Apr 25, 2024

dvd101x commented Apr 27, 2024

josdejong left a comment

dvd101x commented May 12, 2024

Nearly equal with relative and absolute tolerance #3152

Are you sure you want to change the base?

Nearly equal with relative and absolute tolerance #3152

Conversation

dvd101x commented Feb 10, 2024 • edited

josdejong commented Feb 14, 2024

dvd101x commented Feb 18, 2024 • edited

dvd101x commented Feb 18, 2024

josdejong commented Feb 21, 2024

dvd101x commented Feb 26, 2024

josdejong commented Feb 29, 2024

dvd101x commented Feb 29, 2024

dvd101x commented Apr 5, 2024

josdejong commented Apr 5, 2024

dvd101x commented Apr 6, 2024

Appendix

josdejong commented Apr 6, 2024

dvd101x commented Apr 6, 2024

josdejong commented Apr 10, 2024

dvd101x commented Apr 11, 2024

dvd101x commented Apr 12, 2024

josdejong commented Apr 18, 2024

josdejong left a comment

Choose a reason for hiding this comment

josdejong Apr 24, 2024

Choose a reason for hiding this comment

dvd101x Apr 27, 2024 • edited

Choose a reason for hiding this comment

josdejong Apr 29, 2024

Choose a reason for hiding this comment

dvd101x May 12, 2024

Choose a reason for hiding this comment

josdejong Apr 24, 2024

Choose a reason for hiding this comment

dvd101x Apr 27, 2024 • edited

Choose a reason for hiding this comment

josdejong Apr 29, 2024

Choose a reason for hiding this comment

dvd101x May 4, 2024

Choose a reason for hiding this comment

dvd101x May 12, 2024

Choose a reason for hiding this comment

dvd101x commented Apr 25, 2024

dvd101x commented Apr 27, 2024

josdejong left a comment

Choose a reason for hiding this comment

dvd101x commented May 12, 2024

dvd101x commented Feb 10, 2024 •

edited

dvd101x commented Feb 18, 2024 •

edited

dvd101x Apr 27, 2024 •

edited

dvd101x Apr 27, 2024 •

edited