Request: Add additional security and other enterprise features

It is common in enterprise environments that simple open-source tools and workflows fall a little short on meeting a few common requirements.

Typically, this includes (relevant to this tool):

Never pulling anything directly from internet during CI workflows
- Any open-source GIT repos get forked to github enterprise or gitlab etc
- All package manager access get proxied and cached through JFrog Artifactory, Sonatype Nexus, Gitlab, etc.
Never trust any public source download without explicit/manual checksum declaration and verification

This can cause pain for build engineers who need to use a wide range of tools (like this one). Many old tools are simply deemed unusable, or sometimes get forked and sources modified to be used in these environments. Either case is a bummer.

Many modern tools are in a better position to support these things first class, and many do. In this case, there are a few trivial things which could be done if you are interested in supporting such cases:

Support overriding the URL's for all downloads
Support providing a SHA which gets validated right after download
In both cases supporting these values in all of Config, CLI Argument, and EnvVar would be ideal

Here is a pretty robust and mature example of how you could do it, taken from the Conan package manager:

https://github.com/conan-io/conan-center-index/blob/master/recipes/cpython/all/conandata.yml

Of note, conandata.yml only has deal with URL's for one "package" at a time because each package gets it's own conandata.yml. For this tool, you download (and thus need to describe) URL's and shas for multiple "packages", so perhaps it makes sense for your data structure to be different.

You could still choose to support the monolithic single-file-yaml data structure, keyed entirely based on the python version (like in the Conan example). Or, you could support a simpler structure like this, and force the user to use different configs per-version like this:

configs/portable_python_3_9_12.yml

source_urls:
  cpython:
    url: https://my_local_server/Python3_9_12.tar.gz
    sha: "...."
  zlib:
    url: https://my_local_server/ZLib1_2_12.tar.gz
    sha: "...."

configs/portable_python_3_10_4.yml

source_urls:
  cpython:
    url: https://my_local_server/cpython-3_10_4.tar.gz
    sha: "...."
  zlib:
    url: https://my_local_server/zlib-1_2_12.tar.gz
    sha: "...."

I'm not sure which is better but I'd be extremely happy with either.

The good news is that either case seems fairly trivial to support, because you already have robust handling in place for the config file and it looks super easy to add overriding of the URL's thanks to your clean code.

Please let me know what you think about the prospect of this?

I started with shas, but it was a lot of "maintenance" :) especially when iterating initially on the code
I even looked at what files cpython source releases provide, and there is a .asc file associated with each release containing a PGP signature - was going to learn how to leverage that as well.

So yes, very open to adding support for this, to whoever would like to maintain a big url+sha config file like that :)

The code is such that can be added relatively easily down the road.

Great!

To be clear, I don't think this project needs to maintain a fully populated config file like that. Instead, just implement all the conditional variable substitution and conditional verification of sha signature, and then enterprise users can add the file with the values they care about as-needed. From a unit test perspective, just need to assert that when a URL is defined, it's used instead of the hard-coded ones, and if a SHA is defined, the sha comparison function is called.

Does that make sense?

Finally, would you be willing to implement this, or only to review and accept PR?

Open to both :) I'll gladly take a swab at it, I also don't like unchecked downloads...

What do you think about this draft: https://github.com/codrsquad/portable-python/wiki/Configuring-download-urls

Looks great! I would drop the .sum support and give it a go.

The next (and more complicated) thing you might be asked to add will likely be authentication support. Often, internal web/file services like the ones I mentioned disallow anonymous downloads, even for pure open-source forks and binaries inside firewall. Also, more people moving to cloud-hosted services, where allowing anonymous anything is never advisable.

The cheapest way for you to support this would be to support environment variable substitution in the yaml file. Many of these services still support https://${user}:${pass}@service notation. So, by supporting env vars, you can basically punt on any special handling of credentials in your program.

There are lots of syntactical strategies different tools have supported, but ${} is basically standard.

If and when someone wants to support a different web service which doesn't support URL-based creds, the most likely thing it will need is a custom header. In that case, you COULD theoretically add another yaml property called http_headers where people could describe the header in yaml:

source_urls:
  cpython:
    url: https://my_local_server/cpython-3_10_4.tar.gz
    sha: "...."
    http_headers:
        Authorization: "Basic ${BASIC_AUTH_HASH_ENV_VAR}"
  zlib:
    url: https://my_local_server/zlib-1_2_12.tar.gz
    sha: "...."
    http_headers:
        Authorization: "Bearer ${BEARER_TOKEN_ENV_VAR}"

As I said, both env var support and authentication support are extra, but I wanted to give you some ideas no in case it comes up in the near future.

Any progress on this one?

Unfortunately didn't get much time to look at this, may find some time this quarter

I just add proxies supported in download #46 .

@zsimic

@zsimic

With #59 and #60, users can now effectively override all critical aspects of the download and build process. The only other major thing which might be needed in the future which I did not add was overriding environment variables.

Once again, I must mention that the design choices and quality of the code made modifying this project easy and straightforward, so many thanks for that.

For posterity, here is our fully overridden config file which makes use of the new features in both PRs. It downloads all the sources for all the possible modules from an internal mirror (in this case, Github Enterprise). Note it downloads from both tags and github release assets, which are different for some projects. Also note the custom headers for auth and content type, and also note that the URL and header fields both support environment variable string interpolation for auth purposes.

It also patches the configure script for cpython after download, and overrides the configure flags for openssl.

linux:
  allowed-system-libs: /lib/.*

cpython-modules: bzip2 libffi openssl sqlite uuid xz zlib
cpython-url: https://mygitcorp/api/v3/repos/opensource/cpython/tarball/v$version
cpython-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
# Fix a linker issue preventing sqlite3 static linking on linux
# https://github.com/python/cpython/issues/128321
cpython-patches:
  - file: configure
    regex: '  LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS"'
    replacement: '  LIBS="$LIBSQLITE3_LIBS $LIBS"'
  - file: configure.ac
    regex: '  LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS"'
    replacement: '  LIBS="$LIBSQLITE3_LIBS $LIBS"'

bzip2-version: 1.0.8
bzip2-url: https://mygitcorp/api/v3/repos/opensource/bzip2/tarball/bzip2-$version
bzip2-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
libffi-version: 3.4.6
# The .tar.gz in projects public releases has additional files not present the tarball of the git tag
# Note that downloading from release assets requires the Accept header to be set to application/octet-stream
libffi-url: https://mygitcorp/api/v3/repos/opensource/libffi/releases/assets/48150
libffi-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
    Accept: application/octet-stream
openssl-version: 3.0.15
openssl-url: https://mygitcorp/api/v3/repos/opensource/openssl/tarball/openssl-$version
openssl-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
openssl-configure: -v --openssldir=/etc/ssl no-shared no-idea no-tests no-dso
sqlite-version: 3.47.0
sqlite-url: https://mygitcorp/api/v3/repos/opensource/sqlite/tarball/version-$version
sqlite-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
uuid-version: 1.0.3
uuid-url: https://mygitcorp/api/v3/repos/opensource/libuuid/releases/assets/48151
uuid-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
    Accept: application/octet-stream
xz-version: 5.2.5
xz-url: https://mygitcorp/api/v3/repos/opensource/xz/tarball/v$version
xz-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}
zlib-version: 1.2.11
zlib-url: https://mygitcorp/api/v3/repos/opensource/zlib/tarball/v$version
zlib-http-headers:
  - Authorization: Bearer ${GITHUB_ENTERPRISE_AUTH_TOKEN}

Thank you for this awesome contrib! Release with portable-python v1.9.8

Navigation Menu

Repository navigation

Request: Add additional security and other enterprise features #7

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Request: Add additional security and other enterprise features #7

Description

Activity

zsimic commented on Jun 17, 2022

solvingj commented on Jun 17, 2022

zsimic commented on Jun 17, 2022

solvingj commented on Jun 17, 2022

solvingj commented on Mar 20, 2023

zsimic commented on Jan 31, 2024

grammy-jiang commented on Apr 8, 2024

solvingj commented on Feb 6, 2025

zsimic commented on Feb 14, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions