Fixes significant figure calcs for small numbers #440
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #437
The previous code incorrectly assumed that numbers right of the decimal point when
0 < x < 1
were significant. For example,0.0012301
would calculate as7
significant figures. According to various literature (example), because the first two zeros are not involved in measurement decisions they can be simplified as to how it is represented. In the case above we could simplify to1.2301 x 10^-3
consequently requiring only5
significant figures to represent.Because Rust Decimal implicitly stores numbers like this beneath the surface (e.g. above would be stored as
12301 x 10^-7
) we can easily extract the significant figures by relying on the mantissa only.The logic for SF rounding currently leverages the difference between current sf and requested sf to perform calculation. Using the same example, if we were trying to round to 3 significant figures then previously we would try to adjust by
7 - 3 = 4
decimal points which would result in0.001
(1 x 10^-3
) i.e. it assumes that the0
s are significant.The new code aligns with the literature and instead ignores the leading
0
s. The dp adjustment used is5 - 3 = 2
which consequently gives the result0.00123
(123 x 10^-5
) i.e. the leading0
s are not significant.