Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scaling factor should support fractional values #243

Open
That3Percent opened this issue May 20, 2022 · 1 comment
Open

Scaling factor should support fractional values #243

That3Percent opened this issue May 20, 2022 · 1 comment

Comments

@That3Percent
Copy link

I see that from #19 the decision was made to match the Perl implementation by truncating integers. This was perhaps in part due to an "accumulation of error" from #18

But, the documentation specifies that

For example, if you have 23.4 as a sample count you can upscale it to 234, then set factor to 0.1

Which implies that decimal points are supported.

In firestorm the units are in nanoseconds. At least on my computer, the minimum resolution for a sample is 100ns. To "unclutter" the text, I would like to display microseconds instead with fractional values. Eg: instead of 17,100ns, show 17.1μs. This would require fractional values to do well. The current behavior sometimes ends up truncating values to 0.

The accumulation of error can be avoided by not using floating point for calculations throughout, but instead only use the factor at the last moment when formatting the SVG text.

@jonhoo
Copy link
Owner

jonhoo commented May 21, 2022

That seems reasonable to me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants