Repository navigation
Cabal/Haskell: Importing generated path module leads to large closure #164630
Description
Describe the bug
When an executable uses the "paths" module exported by a library (due to the data-dir or data-files directive in the library's cabal file), the closure of that executable is very large. It includes packages such as GHC and its documentation.
Steps To Reproduce
Steps to reproduce the behavior:
- Clone the repo https://github.com/m4dc4p/nix-cabal-data-dir-surprise
- Run
nix-store --query -R --include-outputs $(nix-build -A app). - Notice closure is small & GHC does not appear.
- Run
nix-store --query -R --include-outputs $(nix-build -A big-app). - Closure has many more entries, including GHC (/nix/store/...-ghc-8.10.7) and its documentation (
/nix/store/...-ghc-8.10.7-doc)
Expected behavior
I would not expect that using the exported "paths" module from a library would make the closure so much larger. This is pretty impactful when building docker images. In the above repo, the docker image for big-app is 425MB, while the docker image for app is only 24M:
$ ls -lah $(nix-build -A docker-big-app)
-r--r--r-- 1 justin.bailey admin 475M Dec 31 1969 /nix/store/0bvvq9sdw9bwiqbwyaidxj0wwkjmdif5-docker-image-docker-big-app.tar.gz
$ ls -lah $(nix-build -A docker-app)
-r--r--r-- 1 justin.bailey admin 26M Dec 31 1969 /nix/store/kbs7i25jnnlfcd0i9wxq68q3i829323m-docker-image-docker-app.tar.gz
Additional context
app/Main.hs conditionally imports the Paths_library module from the library package. If the module is imported and used, the closure blows up. This is due to a dependency found in the package.conf.d directory installed with library.
$ nix-store --query -R --include-outputs $(nix-build -A big-app) | grep ghc-8.10.7
/nix/store/4ycwcc6rmyx2bchr7zbigfiyklqdkfik-ghc-8.10.7-doc
/nix/store/qmcdw0bq6590082f3g6sqflqvlyvap5g-ghc-8.10.7
$ nix why-depends --precise $(nix-build -A big-app) /nix/store/4ycwcc6rmyx2bchr7zbigfiyklqdkfik-ghc-8.10.7-doc
/nix/store/j17x9cwaj4japr0mwk8nqvjgx7i6aysf-app-0.1.0.0
└───bin/app: …rld..library_datadir./nix/store/jkndj3lpq6256xyrx2miqglllpqiv4sm-library-0.1.0.0/share/x86_64-os…
→ /nix/store/jkndj3lpq6256xyrx2miqglllpqiv4sm-library-0.1.0.0
└───lib/ghc-8.10.7/package.conf.d/library-0.1.0.0-4DahppUPAwH8P1w4JjXODY.conf: …dock-interfaces:. /nix/store/qlcrbcy2la79hkz81wggdrwh720k7yhj-library-0.1.0.0-doc/share/doc/l…
→ /nix/store/qlcrbcy2la79hkz81wggdrwh720k7yhj-library-0.1.0.0-doc
└───share/doc/library-0.1.0.0/html/Library.html: …> :: <a href="file:///nix/store/4ycwcc6rmyx2bchr7zbigfiyklqdkfik-ghc-8.10.7-doc/share/doc/ghc/ht…
→ /nix/store/4ycwcc6rmyx2bchr7zbigfiyklqdkfik-ghc-8.10.7-doc
The app executable does not have any dependency on GHC:
$ nix-store --query -R --include-outputs $(nix-build -A app) | grep ghc-8.10.7
< no output >
$
Note that the app and big-app derivations compile the same source, but big-app passes a CPP flag so that the Paths_library modules gets imported and used in app\Main.hs.
This bug has come up before, though not with the data-dir aspect, as NixOS/cabal2nix#539 and #155924.
Notify maintainers
Metadata
- system: `"x86_64-darwin"`
- host os: `Darwin 21.3.0, macOS 10.16`
- multi-user?: `no`
- sandbox: `no`
- version: `nix-env (Nix) 2.6.1`
- channels(justin.bailey): `"nixpkgs, nixpkgs-21.11-darwin-21.11-darwin"`
- nixpkgs: `/Users/justin.bailey/.nix-defexpr/channels/nixpkgs`
-
(Current nixpkgs revision is 3e644bd)
Activity
[-]Cabal/Haskell: Importing exports Use of `data-dir` leads to large closure[/-][+]Cabal/Haskell: Importing generated path module leads to large closure[/+]m4dc4p commentedon Mar 18, 2022
Setting
enableSeparateDataOutputtotruefixes the problem!m4dc4p commentedon Mar 18, 2022
Strangely, changing
data-dirtodata-files(w/o theenableSeparateDataOutputflag) also fixes the problem. 🤷maralorn commentedon Mar 18, 2022
I agree with the problem description.
I am sadly not sure if this issue is actionable. i.e. what could we change to fix this without breaking anything else?
m4dc4p commentedon Mar 18, 2022
I thought setting
enableSeparateDataOutputtotruemight do it, but just learned that breaks the build if there is no data files defined.Could cabal2nix be smart enough to recognize when
data-dirordata-filesare used and set the flag appropriately?sternenseemann commentedon Mar 18, 2022
It does.
You should always set
data-files, in fact it's the primary way of doing data file distribution in cabal.data-diris just the place where Cabal looks fordata-files. More specifically, settingdata-dir, but having an emptydata-filesdoes nothing.That a reference to GHC is generated seems to be a weird edge case (maybe related to the fact that a
Paths_module is re-exposed?), I'll have to look into more.lf- commentedon Oct 5, 2022
enableSeparateDataOutput is not seeming to solve the closure size issue in my case, where I am also not even using the Paths_* modules (note that I have remove-references-to in place, and forgot to turn it off, but the references were there in the first place):
It's notable that warp-3.3.21-data has nothing in it, so I'm really not sure what's going on there.
edrex commentedon Nov 7, 2022
It looks as though this PR should allow replacing those paths imports for programs that just want to report version info for various dependencies/self: haskell/cabal#8534
It is in need of qualified reviewers (which I am not).
Atemu commentedon Nov 15, 2023
So, I'm still hitting this issue with git-annex. Exploring the
nix-treerevealed that almost every package has a 2.something GiB closure.maralorn commentedon Nov 15, 2023
Yeah, this is always the same issue, and there is no general fix for this:
This means that the binary contains a reference to the store path of the library, which contains references to a lot of stuff, including GHC.
Possible solutions:
Current workaround:
stringson the binary to figure out which paths are referenced, confirm that it is this issue, by seeing the usual pattern of 6 times the same store path, as above in this thread.postInstallphase that usesremove-references-toon every found store path to replace the store hash with e's.disallowedRequisites = [pkgs.ghc]to the derivation to detect regressions.The advantage of all solutions against the workaround is that all three have the possibility to actually figure out at compile time whether the store paths we don’t want referenced are save to remove.
Atemu commentedon Nov 15, 2023
We can always remove such hacks again. I think it'd be worth trying out after the release.
Is that something each library must choose to do individually?
Could we do that by default on all haskellPackages?
I checked again and all of git-annex' direct dependencies have a direct dependence on GHC. I would not be surprised if this issue affected >90% of haskellPackages.
maralorn commentedon Nov 15, 2023
Yes, sadly.
No, because it means one remove-references-to call for every unwanted dependency. I mean we could theoretically "grep through" the resulting binary to somehow detect these cases, but that feels terribly brittle, and because it can theoretically cause runtime errors, this workaround should be opt-in.
Maybe you are misunderstanding the problem, maybe you don’t. Either way, I am gonna explain:
git-annex, like every package, has build-time dependencies and runtime dependencies. Nix tracks those independently. Build-time dependencies are (the runtime closure of) all store paths mentioned in the .drv. Runtime dependencies are all build-time dependencies that are mentioned in an output of the build.
git-annex, like basically every Haskell package, has a lot of Haskell libraries as build-time dependencies. Generally, all Haskell libraries have a huge runtime closure because they reference all their dependencies and our ghc package, which is quite huge. Thus, all of git-annex direct dependencies having a huge closure is not the core issue here. It is to be expected.
The aim of justStaticExecutables is to create a binary that has none of its Haskell build-time dependencies as runtime dependencies. Thus the issue is, that git-annex has direct runtime dependencies in the first place. That is (probably) caused by the described bug here and can be solved like described above, where the solutions 2. and 3. would fix the problem on the side of all libraries in the git-annex closure that cause the bug (by preventing store-path references to itself in the compiled library), where solution 1 and the workaround fix the problem by preventing the inclusion of that references in the resulting git-annex binary. Thus the workaround can be applied locally in the git-annex package.
maralorn commentedon Nov 15, 2023
Hm, after rereading your comment and my message, my message does not feel as clear, as it could be.
You cannot detect whether a Haskell library triggers this bug by looking at its runtime closure. As I said, the closure is expected to be huge. A Haskell library triggers this bug, if it contains a store-path reference to itself in its compilation output. (A library might also contain self-references in other parts of its outputs, e.g., in the docs, but that’s not an issue because those don’t get linked into the final binary.) I assume you could try to figure that out with the
stringsutility, but I have never tried.haskell.lib.compose.justStaticExecutables: Forbid references to GHC