Repository navigation
toJSON truncates floats #5733
Description
Describe the bug
toJSON unnecessarily truncates floats to six significant figures.
Steps To Reproduce
nix-repl> builtins.toJSON 1.000001
"1"
nix-repl> builtins.toJSON 1048576.1
"1.04858e+06"
Expected behavior
nix-repl> builtins.toJSON 1.000001
"1.000001"
nix-repl> builtins.toJSON 1048576.1
"1048576.1"
nix-env --version output
tested 2.4 and 2.5pre20211126_55275fc
Additional context
found while investigating NixOS/nixpkgs#148824
Activity
mopleen commentedon Dec 7, 2021
This is the place https://github.com/NixOS/nix/blob/master/src/libutil/json.cc#L34
Could be fixed like this
str << std::setprecision(9) << n;but maybe it's better to usefmtor boost formatting to output floating point numbers 🤔pennae commentedon Mar 11, 2022
this also affect
toStringin a different way.toStringalways produces six fractional digits, rounding small floats like 1e-9 to 0.flokli commentedon Jan 25, 2023
This can't be changed, because it would change hashes across Nix versions.
srhb commentedon Apr 25, 2023
According to my bisect, this was "fixed" in #7313 despite the hash changes. We had some scary math issues as a result.
roberth commentedon Apr 30, 2023
I'd like a more configurable
toJSON2primop that allows the precision to be extended, or optionally disables the collapsing of"outPath"fields.flokli commentedon Apr 30, 2023
Displaying floats in decimal is apparently pretty complicated science, and there's multiple competing algorithms to come up with the representation, and even more ways to format it.
It's not entirely well-defined how Nix does it either. If you're curious, check the commit message of https://cs.tvl.fyi/depot/-/commit/1facd889bba724cf20ea14422ee1e57440b3e761, which describes the challenges while trying to mimic Nix as much as possible.
I personally think supporting floats to be printed as strings (and by this, allowing that representation to be leaked into output hashing) was a mistake.
I don't think we should add more builtins treating this differently, or allowing to configure the behavior, but rather discourage the usage of floats in output alltogether. We can't remove it from the language, (even though nixpkgs uses it very little), but we at least could print warnings whenever someone converts a float to string, so they know they produce output that might differ across implementations / architectures / compiler versions.
roberth commentedon Apr 30, 2023
I agree.
Until we explicitly formalize and test the semantics of non-integral numbers in the Nix language, the current implementation ("whatever some platform code does") should be avoided.
Such a project should also include the removal of native floating point operations
It's a non-trivial project to add proper "float" support to Nix.
Nixpkgs/NixOS could sidestep the Nix floats by postprocessing something like
mkFloat = value: { _type = "float"; inherit value;}etc. It already postprocesses thetoJSONoutput in a derivation (inpkgs.formats), so that seems within reason.That also solves the precision limitations, as long as users don't need to compute with floats.
andersk commentedon Sep 10, 2023
Some prior art for deterministically specifying float ↔ string conversions can be found in the ECMAScript specification: Number::toString, StringNumericValue. For historical reasons, that specification has two points where an implementation-defined choice can be made:
In Number::toString step 5, the implementation may choose between multiple representations of the shortest length that all round-trip to the input. A deterministic specification of the best choice is provided in Note 2.
In the RoundMVResult helper for StringNumericValue, the implementation may round excessively-precise inputs up or down to 20 significant digits. A deterministic specification would omit this allowance and use the input as provided.
The Ryū, Grisu-Exact, and Dragonbox algorithms all satisfy this stricter deterministic specification; their difference is in performance and not observable rounding behavior.