[UX bug] dotnet tool dirty state causes paket to install twice or not at all

Description

The build can end up in a bad state when using paket from dotnet tool.

dotnet tool install runs twice, second time around it exists with 1 failing the build.

This happens in docker builds too.

Details:

/usr/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/tmp/tmpWznPq1.tmp.binlog src/Logary.sln
  Installing paket.bootstrapper
  You can invoke the tool using the following command: paket.bootstrapper
  Tool 'paket.bootstrapper' (version '5.180.414') was successfully installed.
  Running paket.bootstrapper to install paket
  Checking Paket version (downloading latest stable)...
  Starting download from https://github.com/fsprojects/Paket/releases/download/5.195.7/Paket.5.195.7.nupkg
  Running dotnet tool install paket --version 5.195.7 --tool-path "/build/.paket" --configfile "/tmp/tmpvOxVqQ.tmp"
  Tool 'paket' is already installed.
/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 1.

Build FAILED.

/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 1.
    0 Warning(s)
    1 Error(s)

Another error, after erasing .paket/.store:

➜  logary ☁️ :(gke_voi-stage_europe-west3_voi-stage-cluster) 🐐:(master) ✗ ./fake.sh build
+ dotnet tool install --tool-path /Users/h/dev/qv2/logary/.paket paket.bootstrapper
error NU1212: Invalid project-package combination for paket.bootstrapper 5.164.0. DotnetToolReference project style can only contain references of the DotnetTool type
The tool package could not be restored.
Tool 'paket.bootstrapper' failed to install. This failure may have been caused by:

* You are attempting to install a preview release and did not use the --version option to specify the version.
* A package by this name was found, but it was not a .NET Core tool.
* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
* You mistyped the name of the tool.

After erasing ~/.dotnet:

/Users/h/dev/qv2/logary> "/usr/local/share/dotnet/dotnet" build src/Logary.sln --configuration Release /bl:/var/folders/t8/zgc3yjy16m5_c7g4s0knbdx80000gn/T/tmp1mV737.tmp.binlog (In: false, Out: false, Err: false)
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/local/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/var/folders/t8/zgc3yjy16m5_c7g4s0knbdx80000gn/T/tmp1mV737.tmp.binlog src/Logary.sln
  Paket ready, checking version:
  The application to execute does not exist: '/Users/h/dev/qv2/logary/.paket/.store/paket/5.195.7/paket/5.195.7/tools/netcoreapp2.1/any/paket.dll'.
/Users/h/dev/qv2/logary/.paket/paket.bootstrapper.proj(37,9): error MSB3073: The command "/Users/h/dev/qv2/logary/.paket/paket --version" exited with code 154.

Build FAILED.

/Users/h/dev/qv2/logary/.paket/paket.bootstrapper.proj(37,9): error MSB3073: The command "/Users/h/dev/qv2/logary/.paket/paket --version" exited with code 154.
    0 Warning(s)
    1 Error(s)

/build> "/usr/bin/dotnet" build src/Logary.sln --configuration Release /bl:/tmp/tmpD1rPpU.tmp.binlog (In: false, Out: false, Err: false)
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/tmp/tmpD1rPpU.tmp.binlog src/Logary.sln
  Installing paket.bootstrapper
  Tool 'paket.bootstrapper' is already installed.
  Running paket.bootstrapper to install paket
  /bin/sh: 2: /tmp/tmpc8eec127c1cb45e4b6ff041ace976e71.exec.cmd: /build/.paket/paket.bootstrapper: not found
/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 127.

Expected behavior

Paket should not crash so much.

Actual behavior

It does crash.

Known workarounds

git clean -fxd && ./fake.sh build

d31b312e reproduces nicely when running make image

I think the solution to this is very careful ignoring of what's in .paket/, namely, paket{,.bootstrapper{,.exe}},.store/

So the only thing this issue showcases is the friction people will experience and the lack of docs on what to ignore and how to use paket with docker builds.

It's not a "bug" per-se but a rather obtuse experience.

What can we do here to improve the docs?

You can document what should be ignored when using Paket with Dockerfiles
You can document where .net tool installs it
You can document that there's no need for either paket nor paket bootstrapper to be present in the source code tree any more since they're automatically downloaded
You can document that paket is now a per-platform binary that must be cleaned before switching platforms
You could document the proper naming cross platform (you've chosen paket.exe vs paket)
You could fix and/or document the insistent (600x of them) warnings about LANG=C not existing (when paket runs)

That clang warning - any ideas how to fix it? Am Di., 29. Jan. 2019, 08:10 hat Henrik Feldt <notifications@github.com> geschrieben:

…

1. You can document what should be ignored when using Paket with Dockerfiles 2. You can document where .net tool installs it 3. You can document that there's no need for either paket nor paket bootstrapper to be present in the source code tree any more since they're automatically downloaded 4. You can document that paket is now a per-platform binary that must be cleaned before switching platforms 5. You could document the proper naming cross platform (you've chosen paket.exe vs paket) 6. You could fix and/or document the insistent (600x of them) warnings about LANG=C not existing (when paket runs) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNBQAZIcEkb5nA7FWYiJvDVE6UHMXks5vH_P4gaJpZM4aUUw4> .

I've tried forcing LANG with LC_ALL and LANG in the container, but the problem is that the compiler or paket sets another LANG environment variable before calling the compiler/dep resolver. The solution is probably to install en_US on the container, but in this instance my google-fu failed me and I failed to find how exactly to do that. The systematic way to do that is to see what container base image MSFT uses for the .net core image and search its documentation for both the method and packages required, and then replace C in my Dockerfile with en_US, while documenting it online in Paket's docs, perhaps with a sample Dockerfile to go with it.

I don't think it's only a container issue. I see it in our linux builds as well (outside of containers)

Yes I see this on some CI servers with fake as well. I have started several attempts at this to change the environment but couldn't fix this yet (I think gitlab shows this iirc)

If you'd find a way to install en_US properly, then perl would probably find it and not complain about it, no matter the linux.

Re: LANG warning:

It's reproducible with MSBuild and this project:

<Project>
  <Target Name="Build">
    <Exec StandardOutputImportance="Low" Command="shasum $(MSBuildThisFile)" ConsoleToMSBuild='true'>
      <Output TaskParameter="ConsoleOutput" PropertyName="Hash" />
    </Exec>
  </Target>
</Project>

I tested this on GitLab CI with this .gitlab-ci.yml:

image: microsoft/dotnet:2.1-sdk

test:
  script: dotnet msbuild -t:Build test.proj

I reduced down what Paket.Restore.targets does to determine if we should perform a restore or not. It has something to do with MSBuilds Exec-Task and how it invokes programs. And apparently shasum is written in Perl and thus we see a Perl warning.

MSBuild sets LANG and LC_ALL to en_US.UTF-8 before invoking programs on Unix: https://github.com/Microsoft/msbuild/blob/f136edbfecec171d73cb4c7fe1c936f6b2bf60a5/src/Tasks/Exec.cs#L597

This gets written to a temporary shell script, which MSBuild actually invokes.

wow very nice hunting. Any idea how to fix/workaround?

Short term: no idea. Some thoughts:

While digging in the SDKs targets files, I found a Hash task. The GenerateAssemblyInfo target (or its CreateGeneratedAssemblyInfoInputsCacheFile sub target) uses it, but I didn't find it in the docs.
There's a new GetFileHash task, but apparently it's not supposed to be used until MSBuild 16.0 comes out.
~~We could add a custom Hash task to Paket.Restore.targets using C# code, like in this example: http://blog.ylett.com/2012/05/msbuild-task-to-generate-hash-of-files.html~~

Ignore that: PaketRestoreLockFileHash = PaketRestoreCachedHash slow #2796 (comment)

We can suppress the warnings by redirecting stderr of shasum to /dev/null, like this:

shasum file.txt 2> /dev/null

That shasum thing should be solved with #3608

Navigation Menu

Repository navigation

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

Description

Details:

Expected behavior

Actual behavior

Known workarounds

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

Description

Description

Details:

Expected behavior

Actual behavior

Known workarounds

Activity

haf commented on Jan 27, 2019

haf commented on Jan 27, 2019

haf commented on Jan 28, 2019

forki commented on Jan 29, 2019

haf commented on Jan 29, 2019

forki commented on Jan 29, 2019

haf commented on Jan 29, 2019

forki commented on Jan 29, 2019

matthid commented on Jan 29, 2019

haf commented on Jan 29, 2019

inosik commented on Feb 5, 2019

inosik commented on Feb 5, 2019

matthid commented on Feb 5, 2019

inosik commented on Feb 7, 2019

inosik commented on Feb 24, 2019

matthid commented on Jul 2, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions