Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes significant figure calcs for small numbers #440

Merged
merged 1 commit into from
Oct 20, 2021
Merged

Conversation

paupino
Copy link
Owner

@paupino paupino commented Oct 20, 2021

Fixes #437

The previous code incorrectly assumed that numbers right of the decimal point when 0 < x < 1 were significant. For example, 0.0012301 would calculate as 7 significant figures. According to various literature (example), because the first two zeros are not involved in measurement decisions they can be simplified as to how it is represented. In the case above we could simplify to 1.2301 x 10^-3 consequently requiring only 5 significant figures to represent.
Because Rust Decimal implicitly stores numbers like this beneath the surface (e.g. above would be stored as 12301 x 10^-7) we can easily extract the significant figures by relying on the mantissa only.

The logic for SF rounding currently leverages the difference between current sf and requested sf to perform calculation. Using the same example, if we were trying to round to 3 significant figures then previously we would try to adjust by 7 - 3 = 4 decimal points which would result in 0.001 (1 x 10^-3) i.e. it assumes that the 0s are significant.

The new code aligns with the literature and instead ignores the leading 0s. The dp adjustment used is 5 - 3 = 2 which consequently gives the result 0.00123 (123 x 10^-5) i.e. the leading 0s are not significant.

@paupino paupino merged commit 5ff1538 into master Oct 20, 2021
@paupino paupino deleted the issue/437 branch January 11, 2022 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Significant figure rounding calculates SF incorrectly for numbers 0 < x < 1
1 participant