OpenSSL is still mean spirited on latest on MacOS

System information

node version: v10.11.0
npm or yarn version: npm 6.4.1
OS/version/architechture: Darwin x86 18.0.0
Applicable nodegit version: Master (0.23.0-alpha.1)

Trying to build locally (npx run rebuild) libssh2 fails to configure (even with openssl headers linked in the ldflags) 3 times in a row, and then the build just continues.

nodegit git:(generation-cleanup) npx node-gyp build
gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@10.11.0 | darwin | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  ACTION binding_gyp_configureLibssh2_target_configure .
[nodegit] Configuring libssh2.
{ Error: Command failed: /Users/maxkorp/development/personal/nodegit/vendor/libssh2/configure
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-libmbedtls-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:289:12)
    at ChildProcess.emit (events.js:182:13)
    at maybeClose (internal/child_process.js:962:16)
    at Socket.stream.socket.on (internal/child_process.js:381:11)
    at Socket.emit (events.js:182:13)
    at Pipe._handle.close (net.js:606:12)
  killed: false,
  code: 1,
  signal: null,
  cmd:
   '/Users/maxkorp/development/personal/nodegit/vendor/libssh2/configure' }
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-libmbedtls-prefix=PATH
 or --with-wincng on Windows

(node:28790) UnhandledPromiseRejectionWarning: Error: Command failed: /Users/maxkorp/development/personal/nodegit/vendor/libssh2/configure
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-libmbedtls-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:289:12)
    at ChildProcess.emit (events.js:182:13)
    at maybeClose (internal/child_process.js:962:16)
    at Socket.stream.socket.on (internal/child_process.js:381:11)
    at Socket.emit (events.js:182:13)
    at Pipe._handle.close (net.js:606:12)
(node:28790) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:28790) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  CC(target) Release/obj.target/libgit2/vendor/libgit2/src/pack-objects.o
  CC(target) Release/obj.target/libgit2/vendor/libgit2/src/pack.o
  CC(target) Release/obj.target/libgit2/vendor/libgit2/src/parse.o
  CC(target) Release/obj.target/libgit2/vendor/libgit2/src/patch_generate.o
  CC(target) Release/obj.target/libgit2/vendor/libgit2/src/patch_parse.o
../vendor/libgit2/src/patch_parse.c:461:12: warning: unused function
      'parse_number' [-Wunused-function]
static int parse_number(git_off_t *out, git_patch_parse_ctx *ctx)
           ^
1 warning generated.

... and so on

Something is bork.

I installed and linked to openssl via brew.

brew install openssl
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"

This was working pre node-v10 PR.

Ok a bit of research here, it looks like on macOS we explicitly don't copy the env anymore (unexpected but somehow unsurprising) but that prevents system headers passed via LDFLags from working. I wonder if acquireopenssl is not working locally?

#1134 you can see this issue, they seems to solve this problem by npm install nodegit@0.23.0-alpha.1

Yeah that prevents this from coming up before a binary is downloaded, but in the case you are explicitly not downloading a binary this is still breaking.

IIRC, issue with copying the env is that it could not properly detect the right OpenSSL during the configureLibssh2 step. My hunch is that probably something to do with include directories; I didn't spend a lot of time investigating once I figured out that copying the env was causing the builds to fail.

You shouldn't need openssl as a dependency unless you're using Electron for nodegit. Is there a particular reason you need to utilize the system openssl instead of the one provided with node? Or are you looking to change the source of the osx openssl as well?

NodeGit should build fine without overriding openssl.

@maxkorp

@maxkorp After upgrading to OSX 10.14 (Mojave), there may be a problem where XCode Command Line Tools does not install all of the necessary headers. The following command may help:
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

@implausible

@implausible @stevex86 it definitely breaks for me on mojave without explicitly including other headers.

@erikwilson I'll try that and see if that helps, thanks for popping in :)

🙃🙃🙃🙃🙃🙃🙃🙃
🙃 THAT WORKED 🙃
🙃🙃🙃🙃🙃🙃🙃🙃

So what do we actually do about this?

Yeah it looks like it's a missing zlib header that causes the whole thing to go down. HOPEFULLY this should be fixed with an xcode-select bugfix and upgrade

@erikwilson

Thanks to @erikwilson we have this:
https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035623

Search for /usr/include

However, some software may fail to build correctly against the SDK and require macOS headers to be installed in the base system under /usr/include. If you are the maintainer of such software, we encourage you to update your project to work with the SDK or file a bug report for issues that are preventing you from doing so. As a workaround, an extra package is provided which will install the headers to the base system. In a future release, this package will no longer be provided. You can find this package at:

/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

Reiterating:

In a future release, this package will no longer be provided

The world is full of hate.

We'll have to figure out which files are missing (I think zlib but we will have to test that) and bundle them appropriately. I wonder if node also includes their zlib headers like they do with openssl.

Life is pain.

WTF Even as root I cant remove any of the installed files.

As an aside, to clean up that package you have to list out out the files while booted into the os install, save them to a file, and then boot into recovery mode and remove them from there.

Based on this, but with some tweaks: https://wincent.com/wiki/Uninstalling_packages_(.pkg_files)_on_Mac_OS_X

On host:

cd /
pkgutil --only-files --files com.apple.pkg.macOS_SDK_headers_for_macOS_10.14 > ~/pkg-files.txt
pkgutil --only-dirs --files com.apple.pkg.macOS_SDK_headers_for_macOS_10.14 > ~/pkg-dirs.txt

In recovery mode

cd /Volumes/{nameOfMacOSVolume}
cat Users/{username}/pkg-files.txt | tr '\n' '\0' | xargs -n 1 -0 sudo rm
cat Users/{username}/pkg-dirs.txt | tr '\n' '\0' | xargs -n 1 -0 sudo rmdir

Using rmdir will prevent it from deleting dirs that have other files in them

Then back on the host

sudo pkgutil --forget com.apple.pkg.macOS_SDK_headers_for_macOS_10.14

Also I removed just zlib from the package, and that does not appear to cause it to break

Looks like it might just be C standard library stuff :(

grep -h include /usr/local/opt/openssl/include/openssl/* | grep -o '<.*>' | grep -v openssl | sort -u
<assert.h>
<ctype.h>
<errno.h>
<inttypes.h>
<krb5.h>
<limits.h>
<resource.h>
<stdarg.h>
<stddef.h>
<stdint.h>
<stdio.h>
<stdlib.h>
<string.h>
<sys/socket.h>
<sys/time.h>
<sys/timeb.h>
<sys/times.h>
<sys/timeval.h>
<sys/types.h>
<time.h>
<windows.h>
<winsock.h>

@maxkorp

@maxkorp It's still strange that you're trying to use system openssl headers. The install path as of now should never utilize or require brew or your system ssl headers.

The header data should be located in /your_home_dir/.node-gyp/10.11.0/include/node where the header files are supposed to be. Then the link to node is supposed to provide the openssl implementation for those headers. Why are you trying to link against the system in response? I think you might be better off investigating why the provided headers from node for openssl are failing to work when configuring libssh2, than to bind against a system lib. Does that make sense? I hope I'm explaining this clearly.

Yes, there are definitely some spooky things one could do during npm install. ☹️

This is the libssh2_config.h generated on my machine (Mac OS High Sierra):
https://pastebin.com/7usEuVP3

For each of the defines that are turned on, it seems to indicate whether the system has POSIX C headers. Other than that, the interesting ones are HAVE_LIBSSL, HAVE_LIBZ, and LIBSSH2_OPENSSL. Both OpenSSL and zlib are linked into Node. For Electron builds, zlib is provided by Electron.

Since MacOS is a relatively stable environment, and we don't seem to be using any libraries that are not provided, I feel very safe to provide the libssh2_config.h header statically.

I also ran the configure script on my machine against an Electron build and diff'd it against the Node build. No differences.

npm module install scripts are a GIGANTIC attack vector, yes.

So we'd not need to configure on linux then, either?

I suppose not, though if any OS actually needs configure it would be linux. Do we build for Arm and how different is that architecturally from amd64, x86_64? Any chance configuration like big endian vs little endian is more likely to show up on linux than sanctioned win32 / mac os land?

@erikwilson

@erikwilson Does this seem to fix your issues? #1569

Looks like it works to me. Another option might be to disable just the crypto lib check in configure.ac & generate a new configure, assume openssl will be used.

Actually, monkeying around with vendor autoconfig stuff probably isn't a good idea either. I don't know the whole repercussion of the build system using electron and byo openssl. Since Apple appears to hate developers and this is only an issue in Mojave (and likely future OSX versions) for verifying openssl availability, as a pre-configure step we could link to /usr/lib in /tmp (eg ln -s /usr/lib /tmp/usrlib) and then pass LDFLAGS=-L/tmp/usrlib or similar to bypass the use of syslibroot flag.

So we're not planning on moving away from our conan openssl vendor, and I also agree with you that monkeying around in vendor autoconfig stuff probably isn't a good idea. I think we'll roll with our solution. Once that is merged in I am going to close this issue.

Cool, I'm not saying that you should move away from conan openssl, just there might be a solution where you can still configure for systems that need it and fix configuring on Mojave.

#1569 closes this issue. There is also a new prerelease 0.23.0-alpha.2 on the next tag.

Navigation Menu

Repository navigation

OpenSSL is still mean spirited on latest on MacOS #1566

System information

In a future release, this package will no longer be provided

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OpenSSL is still mean spirited on latest on MacOS #1566

Description

System information

Activity

maxkorp commented on Oct 9, 2018

LinxinS97 commented on Oct 9, 2018

maxkorp commented on Oct 9, 2018

emmax86 commented on Oct 10, 2018

implausible commented on Oct 10, 2018

erikwilson commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

In a future release, this package will no longer be provided

maxkorp commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

maxkorp commented on Oct 12, 2018

erikwilson commented on Oct 12, 2018

implausible commented on Oct 12, 2018

26 remaining items

implausible commented on Oct 16, 2018

emmax86 commented on Oct 16, 2018

emmax86 commented on Oct 16, 2018

maxkorp commented on Oct 16, 2018

implausible commented on Oct 16, 2018

implausible commented on Oct 17, 2018

erikwilson commented on Oct 17, 2018

erikwilson commented on Oct 17, 2018

implausible commented on Oct 17, 2018

erikwilson commented on Oct 17, 2018

implausible commented on Oct 19, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions