-
-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add some Json tests #5037 -followup #5070
Conversation
…comment some Json tests
@@ -84,7 +84,7 @@ let rec serialize | |||
else if System.Double.IsPositiveInfinity f then | |||
w.WriteStringValue "Infinity" | |||
else | |||
let result = sprintf "%.12g" f | |||
let result = sprintf "%.16g" f |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changing the float formatting precision to 16 makes it consistent with other languages. Is that sufficient, or do we want to move away from sprintf entirely?
From my reading, most languages seem to not explicitly set float formatting precision; the limitations typically are inherited from the IEEE 754 double precision standard.
some useful links and discussions:
I'm uncertain about our next steps. We can adhere to the IEEE 754 standard, which allows for up to 16 significant digits (link). This aligns with the consistent results we obtained while testing other languages |
Paul's notes:
I've tried using sprintf "%.17g" but I didn't like the result we got from some tests.
We've decided to use sprintf "%.16g" for now and plan to remove it later. |
No changelog
#5037 follow-up
The float formatting precision was increased from %.12 to %.16 to match other programming languages (F#, python, and Javascript)
The added tests that use [number]e+[pow] were removed.