Skip to content

Commit 99d5a68

Browse files
dumoladiroiban
andauthored
Generic Linux ARM64 package. (#143)
* GMP fix for Debian 10 arm64. * Imported latest cryptography sources. * Alpine 3.12 deps update. * Use cryptography 2.9.2. * Build generic Linux on Ubuntu 16.04 arm64/x64. * GCC voodoo for building cryptography through pip against our OpenSSL libs. * Removed cryptography sources from python-modules/. * Changes after own review. * Add travis-ci. * Increased timeouts for GitHub actions. skip-ci * Updated README. * Brought back bits lost when merging with master. * Updated setuptools version. * Refined Homebrew hacking on macOS. * Minor improvements for GitHub actions. * Updated documented deps. [skip-ci] * Added a hack for missing makeinfo, finally. * Link makeinfo to /bin/true to please GMP's build too. * Pin Ubuntu 16.04 on Xenial * Fixed deps checking on ARM64. * Remove bzip2's symlinked binaries to avoid bcrypt botching them. * Changes after own review. * Fixed building from scratch. * Fixed brink.conf sourcing. * Fixed GitHub actions Windows builds. * Changes after review. Co-authored-by: Adi Roiban <adi.roiban@chevah.com>
1 parent 9209c4d commit 99d5a68

11 files changed

Lines changed: 771 additions & 740 deletions

File tree

‎.github/workflows/main.yaml‎

Lines changed: 19 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ jobs:
2323
ubuntu-20-04:
2424
# The type of runner that the job will run on
2525
runs-on: ubuntu-20.04
26-
timeout-minutes: 20
26+
timeout-minutes: 30
2727
# Steps represent a sequence of tasks that will be executed as part of the job
2828
steps:
2929
# Make sure there are no concurrent jobs
@@ -42,13 +42,6 @@ jobs:
4242
build-python-package
4343
key: ${{ runner.os }}-${{ hashFiles('pavement.py') }}
4444

45-
# Do not run when the following strings are used in the commit message.
46-
- name: Fail on skip ci
47-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-ci'
48-
49-
- name: Fail on skip gha
50-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-gha'
51-
5245
- name: Prepare OS
5346
run: sudo apt-get install libncurses5-dev
5447

@@ -72,7 +65,7 @@ jobs:
7265

7366
macos-10-15:
7467
runs-on: macos-10.15
75-
timeout-minutes: 30
68+
timeout-minutes: 60
7669
steps:
7770
- uses: chevah/auto-cancel-redundant-job@v1
7871
- uses: actions/checkout@v2
@@ -86,24 +79,24 @@ jobs:
8679
build-python-package
8780
key: ${{ runner.os }}-${{ hashFiles('pavement.py') }}
8881

89-
- name: Fail on skip ci
90-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-ci'
91-
92-
- name: Fail on skip gha
93-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-gha'
82+
- name: Download needed deps for the detected OS
83+
run: ./brink.sh detect_os
9484

95-
# Avoid linking to Homebrew's libintl. We can use needed tools from /usr/bin.
96-
- name: Prepare OS
85+
# Avoid linking to Homebrew's libintl during build.
86+
# We can use needed tools from /usr/bin.
87+
- name: Hack Homebrew
9788
run: |
98-
sudo find /usr/local -iname 'libintl*' -exec rm -f {} +
89+
sudo find /usr/local -name 'libintl*' -exec chmod a-r {} +
9990
sudo rm -f /usr/local/bin/{wget,curl,git}
10091
101-
- name: Download needed deps for the detected OS
102-
run: ./brink.sh detect_os
103-
10492
- name: Build Python
10593
run: ./chevah_build build
10694

95+
# Fix back Homebrew, to make everything functional.
96+
- name: Unhack Homebrew
97+
run: |
98+
sudo find /usr/local -name 'libintl*' -exec chmod a+r {} +
99+
107100
- name: Test Python
108101
run: ./chevah_build test
109102

@@ -117,7 +110,7 @@ jobs:
117110

118111
windows-2019:
119112
runs-on: windows-2019
120-
timeout-minutes: 25
113+
timeout-minutes: 30
121114
steps:
122115
- uses: chevah/auto-cancel-redundant-job@v1
123116
- uses: actions/checkout@v2
@@ -131,25 +124,20 @@ jobs:
131124
build-python-package
132125
key: ${{ runner.os }}-${{ hashFiles('pavement.py') }}
133126

134-
- name: Fail on skip ci
135-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-ci'
136-
137-
- name: Fail on skip gha
138-
run: git log -1 --pretty=format:"%s" ${{ toJSON(github.sha) }} | grep -v 'skip-gha'
139-
140127
# Add packages needed to build our patched pycrypto.
141128
- name: Prepare OS
142129
run: |
143130
chocolatey install -y make vcpython27
144131
132+
# Explicitly run our scripts with Bash, not PowerShell (GitHub's default).
145133
- name: Download needed deps for the detected OS
146-
run: ./brink.sh detect_os
134+
run: bash ./brink.sh detect_os
147135

148136
- name: Build Python
149-
run: ./chevah_build build
137+
run: bash ./chevah_build build
150138

151139
- name: Test Python
152-
run: ./chevah_build test
140+
run: bash ./chevah_build test
153141

154142
- name: Test Compat
155-
run: ./chevah_build test_compat
143+
run: bash ./chevah_build test_compat

‎.travis.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#
2+
# Experimental Travis-CI
3+
# For now, the build matrix has a single member: linux-arm64
4+
#
5+
# Pinned on the oldest available distribution, to get a generic Linux build.
6+
7+
language: c
8+
9+
arch:
10+
- arm64
11+
12+
os:
13+
- linux
14+
15+
dist:
16+
- xenial
17+
18+
install: skip
19+
20+
script:
21+
- ./brink.sh detect_os
22+
- ./chevah_build build
23+
- ./chevah_build test
24+
- ./chevah_build test_compat

‎README.rst‎

Lines changed: 23 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -12,33 +12,40 @@ please check https://binary.chevah.com/production/python/.
1212

1313
Building steps as used by Buildbot:
1414

15-
* Login to the desired platform.
16-
* Get code from GitHub.
17-
* ```./chevah_build build```
18-
* ```./chevah_build test```
19-
* Upload to binary dist publishing website.
15+
* Login to a system running on the desired platform (e.g. Ubuntu Server 20.04).
16+
* Get the code for this repository from GitHub.
17+
* ``./brink.sh detect_os``
18+
* ``./chevah_build build``
19+
* ``./chevah_build test``
20+
* ``./chevah_build test_compat``
2021

21-
You can try the first 4 steps on your own to build and test a new
22-
Python package on your own.
22+
You can try the above steps on your own to build and test a new Python package.
2323

24-
Use ```./chevah_build help``` to discover all available commands.
24+
To have new Python packages uploaded (typically for all supported platforms),
25+
you should try the following, after securing access to Chevah infrastructure:
26+
27+
* ``./brink.sh test_remote group-all --properties=force_upload_production=yes
28+
29+
Use ``./chevah_build help`` to discover all available commands.
2530

2631

2732
Patching upstream code
2833
----------------------
2934

3035
This repository contains a lot of imported code from upstream repos:
31-
Python, OpenSSL, LibreSSL, SQLite, gmp, libffi, PyCrypto, etc.
36+
Python, OpenSSL, SQLite, gmp, libffi, PyCrypto, etc.
3237

3338
You can find those sources in two sub-directories:
34-
* ```src/```
35-
* ```python-modules/```
39+
40+
* ``src/``
41+
* ``python-modules/``
3642

3743
Local changes to upstream projects are kept separated from upstream code.
38-
For every problem we fix downstream there's a patch to be applied at build
44+
For every problem we fix downstream, there's a patch to be applied at build
3945
time. Those patches are kept in one of these places:
40-
* ```src/$PROJECT/*.patch```
41-
* ```python-modules/$PROJECT-$VERSION-patches/```
46+
47+
* ``src/$PROJECT/*.patch``
48+
* ``python-modules/$PROJECT-$VERSION-patches/``
4249

4350
New patches added as documented above will be picked up automatically
4451
by the build system.
@@ -60,7 +67,7 @@ External dependencies and associated vulnerabilities
6067
This Python package builds on and links to third-party libraries.
6168

6269
To document them for all OS'es and to list their known vulnerabilities,
63-
the LibreOffice spreadsheet ```external_deps.fods``` is used.
70+
the LibreOffice spreadsheet ``external_deps.fods`` is used.
6471

6572
For your convenience, this flat ODS file is also exported in CSV format in
66-
```external_deps.csv```.
73+
``external_deps.csv``.

‎brink.conf‎

Lines changed: 1 addition & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
BASE_REQUIREMENTS='pip==20.1.1 chevah-brink==0.74.2 paver==1.2.4'
1+
BASE_REQUIREMENTS='pip==20.1.1 chevah-brink==0.75 paver==1.2.4'
22
PYTHON_CONFIGURATION='default@2.7.18.4b61bd67:sol10@2.7.8.4b61bd67'
33
BINARY_DIST_URI='http://binary.chevah.com/production'
44
PIP_INDEX='http://pypi.chevah.com/simple'
@@ -7,20 +7,3 @@ PIP_INDEX='http://pypi.chevah.com/simple'
77
# * 'build' is a sub-dir used for building GMP / Python / gmpy / etc.
88
# * $CHEVAH_BUILD_DIR is used by the Python that builds the above.
99
CHEVAH_BUILD_DIR='build-python-package'
10-
11-
# For the Python build, we need to detect current OS and get needed deps.
12-
# This extedens brink.sh with the `detect_os` command option, which writes
13-
# the BUILD_ENV_VARS file in the Python build dir for chevah_build to pick up.
14-
execute_venv() {
15-
set +o nounset
16-
local command=$1
17-
set -o nounset
18-
if [ "$command" == "detect_os" ]; then
19-
echo "PYTHON_VERSION=$PYTHON_NAME" > BUILD_ENV_VARS
20-
echo "OS=$OS" >> BUILD_ENV_VARS
21-
echo "ARCH=$ARCH" >> BUILD_ENV_VARS
22-
exit 0
23-
fi
24-
25-
${PYTHON_BIN} $PYTHON3_CHECK -c 'from paver.tasks import main; main()' "$@"
26-
}

‎brink.sh‎

Lines changed: 31 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -607,11 +607,25 @@ check_os_version() {
607607
# For old unsupported Linux distros (some with no /etc/os-release) and for other
608608
# unsupported Linux distros (eg. Arch), we check if the system is glibc-based.
609609
# If so, we use a generic code path that builds everything statically,
610-
# including OpenSSL, thus only requiring glibc 2.x.
610+
# including OpenSSL, thus only requiring glibc 2.X, where X differs by arch.
611611
#
612612
check_linux_glibc() {
613613
local glibc_version
614614
local glibc_version_array
615+
local supported_glibc2_version
616+
617+
# Supported minimum minor glibc 2.X versions for various arches.
618+
# For x64, we build on SLES 11 with glibc 2.11.3.
619+
# For arm64, we build on Ubuntu 16.04 with glibc 2.23.
620+
# Beware we haven't normalized arch names yet.
621+
case "$ARCH" in
622+
"amd64"|"x86_64"|"x64")
623+
supported_glibc2_version=11
624+
;;
625+
"aarch64"|"arm64")
626+
supported_glibc2_version=23
627+
;;
628+
esac
615629

616630
(>&2 echo -n "Couldn't detect a supported distribution. ")
617631
(>&2 echo "Trying to treat it as generic Linux...")
@@ -630,7 +644,7 @@ check_linux_glibc() {
630644
exit 19
631645
fi
632646

633-
# Parsing tested with glibc 2.11.x and 2.29, eglibc 2.13 and 2.19.
647+
# Tested with glibc 2.5/2.11.3/2.12/2.23/2.28-31 and eglibc 2.13/2.19.
634648
glibc_version=$(ldd --version | head -n 1 | rev | cut -d\ -f1 | rev)
635649

636650
if [[ $glibc_version =~ [^[:digit:]\.] ]]; then
@@ -647,16 +661,17 @@ check_linux_glibc() {
647661
fi
648662

649663
# We pass here because:
650-
# 1. In python-package building should work with older glibc version.
664+
# 1. Building python-package should work with an older glibc version.
651665
# 2. Our generic "lnx" runtime might work with a slightly older glibc 2.
652-
if [ ${glibc_version_array[1]} -lt 11 ]; then
653-
(>&2 echo "Beware glibc versions older than 2.11 were NOT tested!")
654-
(>&2 echo "Detected glibc version: $glibc_version")
666+
if [ ${glibc_version_array[1]} -lt ${supported_glibc2_version} ]; then
667+
(>&2 echo -n "Detected glibc version: ${glibc_version}. Versions older")
668+
(>&2 echo " than 2.${supported_glibc2_version} were NOT tested!")
669+
655670
fi
656671

657672
set -o errexit
658673

659-
# glibc 2 detected, we set $OS for a generic build.
674+
# glibc 2 detected, we set $OS for a generic Linux build.
660675
OS="lnx"
661676
}
662677

@@ -720,8 +735,8 @@ detect_os() {
720735
;;
721736
ubuntu|ubuntu-core)
722737
os_version_raw="$VERSION_ID"
723-
# 12.04/14.04 have OpenSSL 1.0.1, use generic Linux.
724-
check_os_version "$distro_fancy_name" 16.04 \
738+
# For versions with older OpenSSL, use generic build.
739+
check_os_version "$distro_fancy_name" 18.04 \
725740
"$os_version_raw" os_version_chevah
726741
# Only LTS versions are supported. If it doesn't end in
727742
# 04 or first two digits are uneven, use generic build.
@@ -865,6 +880,13 @@ if [ "$COMMAND" = "purge" ] ; then
865880
exit 0
866881
fi
867882

883+
# Initialize BUILD_ENV_VARS file when building python-package from scratch.
884+
if [ "$COMMAND" == "detect_os" ]; then
885+
echo "PYTHON_VERSION=$PYTHON_NAME" > BUILD_ENV_VARS
886+
echo "OS=$OS" >> BUILD_ENV_VARS
887+
echo "ARCH=$ARCH" >> BUILD_ENV_VARS
888+
exit 0
889+
fi
868890

869891
if [ "$COMMAND" = "get_python" ] ; then
870892
OS=$2

0 commit comments

Comments
 (0)