Repository navigation
cross-compilation support for python build #48004
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tarekziade'
closed_at = <Date 2013-02-02.16:51:13.100>
created_at = <Date 2008-09-01.21:49:18.597>
labels = ['type-feature', 'build']
title = 'cross-compilation support for python build'
updated_at = <Date 2013-02-06.13:27:50.032>
user = 'https://bugs.python.org/rpetrov'bugs.python.org fields:
activity = <Date 2013-02-06.13:27:50.032>
actor = 'doko'
assignee = 'tarek'
closed = True
closed_date = <Date 2013-02-02.16:51:13.100>
closer = 'rpetrov'
components = ['Cross-Build']
creation = <Date 2008-09-01.21:49:18.597>
creator = 'rpetrov'
dependencies = []
files = ['17361', '17778', '18625', '19586', '19588', '20534', '20536', '21570', '22036', '23034', '24923', '25762', '25850', '27444', '27896']
hgrepos = []
issue_num = 3754
keywords = ['patch', 'needs review']
message_count = 56.0
messages = ['72299', '77241', '98223', '105859', '108744', '110685', '114776', '127140', '132102', '133252', '136371', '136772', '136885', '136887', '136911', '136962', '136963', '136987', '137001', '142972', '155693', '155971', '155972', '155977', '155980', '156240', '161972', '162421', '162464', '164376', '164378', '164381', '164382', '164385', '164395', '164396', '164402', '164403', '164409', '164412', '164416', '166740', '171787', '172156', '174925', '174926', '174928', '175009', '180590', '180594', '180701', '181180', '181447', '181449', '181457', '181520']
nosy_count = 18.0
nosy_names = ['doko', 'wrobell', 'vapier', 'tarek', 'eric.araujo', 'rpetrov', 'Arfrever', 'devurandom', 'WhiteTiger', 'BreamoreBoy', 'python-dev', 'David.Kern', 'Greg.Hellings', 'kalev', 'marqvar', 'Ray.Donnelly', 'eudoxos', 'ambrop7']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3754'
versions = ['Python 3.3']
Activity
rpetrov commentedon Sep 1, 2008
This is minimal patch that add basic cross-compilation possibilities for
python build (configure script).
The patch add macro AC_CANONICAL_HOST.
This macro require files config.guess, config.sub. The patch don't
include them. You may obtain them from GNU automake tarbal.
As result of macro new variable $host ("host triplet":=cpu-verdor-os) is
used to detect so called "host system".
Since this is basic patch, detection of build system in native builds
based on $ac_sys_system and/or $ac_sys_release isn't replaced. This
detection isn't appropriate for cross-compilation environment as contain
values for "build system" and has to be replaces in addition by future
patches.
Also the patch posted in http://bugs.python.org/issue3718 (about
environment variable MACHDEP) isn't required for native builds, but will
help in case of cross-compilation.
rpetrov commentedon Dec 7, 2008
The support for *-mingw host is enhanced to native msys environment.
This include check for device file /dev/ptmx. The MSYS report that file
exist. As the check is skipped on mingw* host os now cugwin environment
can be used too.
Also check libintl is skipped for mingw as the native build don't use it.
rpetrov commentedon Jan 24, 2010
system python has to be at least revision 77704 (trunk)
rpetrov commentedon May 16, 2010
As part of bpo-8510 (update to autoconf2.65) configure script is modernized and most of updates from patches attached to this issue now are in repository.
Starting from 16 may new patches will include in addition minimal updates of setup.py and Makefile.pre.in (moved from bpo-3871) :
[-]minimal cross-compilation support for configure[/-][+]cross-compilation support for python build[/+]rpetrov commentedon Jun 26, 2010
BreamoreBoy commentedon Jul 18, 2010
Could someone with knowledge of the Python build systems please comment on this, thanks.
rpetrov commentedon Aug 24, 2010
It seems to me I forgot to upload version after updates in Lib/sysconfig.py from bpo-7880.
I could not understand what Florent Xicluna (flox). I could not reproduce it. To me update abspath -> realpath is bogus and with reverted update of Lib/sysconfig.py test_sysconfig pass.
This patch is restored previous behaviour and now symlinked system python into cross build tree will behave as before i.e. as expected ;)
rpetrov commentedon Jan 26, 2011
fixed patch failure on Parser/pgen.stamp
merwok commentedon Mar 25, 2011
I see some “fix for issue #NNN is bogus” in your patch: would you open separate bug reports for those? A diff file is not a very useful way to report bugs or express opinions <wink>.
rpetrov commentedon Apr 7, 2011
Uhh after some pseudo multiarch improvements previous patch fail. So new one is uploaded.
Also with this version cross-build won't build pgen$(EXEEXT).
GregHellings commentedon May 20, 2011
Current patch errors with the following message:
gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -L/manual/lib -L/binary/lib -L/manual/lib -L/binary/lib Parser/acceler.o Parser/grammar1.o Pars
er/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o
Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain
.o -lintl -lpthread -o Parser/pgen.exe
./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
./Grammar/Grammar: line 18: single_input:: command not found
./Grammar/Grammar: line 18: simple_stmt: command not found
./Grammar/Grammar: line 18: compound_stmt: command not found
./Grammar/Grammar: line 19: syntax error near unexpected token
NEWLINE' ./Grammar/Grammar: line 19:file_input: (NEWLINE | stmt)* ENDMARKER'make: *** [Parser/pgen.stamp] Error 2
rpetrov commentedon May 24, 2011
Greg, ensure correct configure script first as run commands autoheader and autoconf.
Updates to configure script are not in patch . The patch include updates to source configure.in and autoconf command will update "configure".
35 remaining items
ambrop7 commentedon Nov 5, 2012
I've applied all patches from py3k-20121004-CROSS.tgz to Python 3.3.0 except 2, 4, and 9 (which didn't apply), but it's not working. After it builds the built in modules, it tries to run the parser generator which was cross-compiled.
make Parser/pgen
make[1]: Entering directory
/root/Python-3.3.0' armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/dynamic_annotations.o](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.o) [Python/dynamic_annotations.c](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/mysnprintf.o](https://github.com/python/cpython/blob/main/Python/mysnprintf.o) [Python/mysnprintf.c](https://github.com/python/cpython/blob/main/Python/mysnprintf.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/pyctype.o](https://github.com/python/cpython/blob/main/Python/pyctype.o) [Python/pyctype.c](https://github.com/python/cpython/blob/main/Python/pyctype.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/tokenizer_pgen.o](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.o) [Parser/tokenizer_pgen.c](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/printgrammar.o](https://github.com/python/cpython/blob/main/Parser/printgrammar.o) [Parser/printgrammar.c](https://github.com/python/cpython/blob/main/Parser/printgrammar.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/parsetok_pgen.o](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.o) [Parser/parsetok_pgen.c](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/pgenmain.o](https://github.com/python/cpython/blob/main/Parser/pgenmain.o) [Parser/pgenmain.c](https://github.com/python/cpython/blob/main/Parser/pgenmain.c) armv5tel-softfloat-linux-gnueabi-gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes [Parser/acceler.o](https://github.com/python/cpython/blob/main/Parser/acceler.o) [Parser/grammar1.o](https://github.com/python/cpython/blob/main/Parser/grammar1.o) [Parser/listnode.o](https://github.com/python/cpython/blob/main/Parser/listnode.o) [Parser/node.o](https://github.com/python/cpython/blob/main/Parser/node.o) [Parser/parser.o](https://github.com/python/cpython/blob/main/Parser/parser.o) [Parser/bitset.o](https://github.com/python/cpython/blob/main/Parser/bitset.o) [Parser/metagrammar.o](https://github.com/python/cpython/blob/main/Parser/metagrammar.o) [Parser/firstsets.o](https://github.com/python/cpython/blob/main/Parser/firstsets.o) [Parser/grammar.o](https://github.com/python/cpython/blob/main/Parser/grammar.o) [Parser/pgen.o](https://github.com/python/cpython/blob/main/Parser/pgen.o) [Objects/obmalloc.o](https://github.com/python/cpython/blob/main/Objects/obmalloc.o) [Python/dynamic_annotations.o](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.o) [Python/mysnprintf.o](https://github.com/python/cpython/blob/main/Python/mysnprintf.o) [Python/pyctype.o](https://github.com/python/cpython/blob/main/Python/pyctype.o) [Parser/tokenizer_pgen.o](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.o) [Parser/printgrammar.o](https://github.com/python/cpython/blob/main/Parser/printgrammar.o) [Parser/parsetok_pgen.o](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.o) [Parser/pgenmain.o](https://github.com/python/cpython/blob/main/Parser/pgenmain.o) -lpthread -ldl -lpthread -lutil -o [Parser/pgen](https://github.com/python/cpython/blob/main/Parser/pgen) make[1]: Leaving directory/root/Python-3.3.0'Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
Parser/pgen: Parser/pgen: cannot execute binary file
make: *** [Include/graminit.h] Error 126
ambrop7 commentedon Nov 5, 2012
Forgot to mention: I did run autoreconf after applying the patches. I'm attaching the full output.
ambrop7 commentedon Nov 5, 2012
A minor issue: if only --host= is specified on command line but not --build=, then cross_compiling variable is only defined after AC_PROG_CC is called. However, configure.ac uses it before that (e.g. the part at the top which looks for a python interpreter). Well, more precisely, something in autoconf sets it to cross_compiling=maybe.
rpetrov commentedon Nov 6, 2012
HI, Ambroz
Ambroz Bizjak wrote:
About patches :
2: restore graminit.* to source directory
4: restore AST to source directory
After this touch files to avoid timestamp dependencies to launch extra
build .
You issue was reported and I guess is documented . It is related to
checkout from repository and to general build process.
I have a shell script to this but in brief script touch in source tree
following files:
If you decide to build in source tree patch 2 and 4 could be skipped .
Note that listed patches allow user to keep separate builds at same time
for different platforms. As side effect part of patches could help to
build with different configuration options.
For one time cross-build users could use following scenario:
Repeat 2 & 3 for each platform
about 9: pass all top configure arguments to libffi configure to allow
users to build it properly
It is save to ignore this patch if libffi build is fine for you host
platform. Otherwise .... just change setup to use only
...
config_args = sysconfig.get_config_var("CONFIG_ARGS")
...
and replace " ".join(config_args) with config_args in command.
For instance I have build for a platform and with filtered "configure
arguments" build fail. Instead to patch libffi code I prefer from main
configure command line to control build of libffi.
Roumen
python-dev commentedon Jan 25, 2013
New changeset 11a18263ceb7 by doko in branch '2.7':
http://hg.python.org/cpython/rev/11a18263ceb7
New changeset e28b30e6eee6 by doko in branch '3.2':
http://hg.python.org/cpython/rev/e28b30e6eee6
New changeset 5464a534e7bd by doko in branch '3.3':
http://hg.python.org/cpython/rev/5464a534e7bd
New changeset ee48728e3695 by doko in branch 'default':
http://hg.python.org/cpython/rev/ee48728e3695
doko42 commentedon Jan 25, 2013
about py3k-20121004-CROSS.tgz:
committed 0001-CROSS-fix-typo-in-thread-AC_CACHE_VAL.patch
0002-CROSS-restore-graminit.-to-source-directory.patch
is this necessary? Assuming that you have correct time
stamps, this is something which usually is not built during
a cross build.
Can we just have a shell script which touches some files
so that you can cross-build even from the repository without
building pgen?
Do the release managers have such a script which is run
when building a release?
0003-CROSS-restore-importlib-header-to-source-directory-a.patch
Same as for 0002, using correct timestamps. The EXEEXT is a
different issue. However I'm not sure what to use, EXEEXT,
or BUILDEXEEXT.
0004-CROSS-restore-AST-to-source-directory.patch
Same as for 0002. Btw, does this fail with a native out-of-tree
build too?
0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch
A comment like "The fix for issue bpo-13150 is bogus and break
everything" might not the best way to convince maintainers ...
This works for me, and IMO should be dropped, or please explain
why this doesn't work for you.
0006-CROSS-initialise-include-and-library-paths.patch
Afaics, a similar patch is now applied. Can be dropped.
0007-CROSS-set-_PYTHON_PROJECT_BASE-to-current-build-dir.patch
Applied.
0008-CROSS-use-_PYTHON_PROJECT_BASE-in-distutils-sysconfi.patch
Applied.
0009-CROSS-pass-all-top-configure-arguments-to-libffi-con.patch
Why is it needed to pass all configure args?
0010-CROSS-warn-only-if-readelf-is-not-in-host-triplet-fo.patch
Why is this needed just for readelf? If you do have a cross
toolchain installed, then this should be available with
the triplet name, same as for gcc, as, ...
0011-CROSS-append-gcc-library-search-paths-instead-to-pre.patch
Looks ok. Could you mention where this does make a difference,
or do you just want to keep the search order as defined by
the compiler?
0012-CROSS-avoid-ncursesw-include-path-hack.patch
This would break cross-building in a multiarch environment.
Maybe we need a --with-curses-incdir=<> option.
0013-CROSS-properly-detect-WINDOW-_flags-for-different-nc.patch
This should be a separate issue. Not cross specific.
0014-CROSS-use-build-directory-as-root-for-regression-tes.patch
why is this necessary?
0015-CROSS-test-tools-has-to-depend-only-from-location-of.patch
Looks ok, but not cross specific. Could you file a separate issue?
0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch
Looks like a separate issue.
rpetrov commentedon Jan 26, 2013
Matthias Klose wrote:
All above is related to build out of source tree.
Try to keep a build tree for long time. Synchronize source, for instance
once in week, and try to rebuild. After updates in those files python
(native build) crash.
Solutions:
a) restore to source tree as before and only for cross build touch files
to avoid regeneration.
b) after synchronization copy to build tree but now in all cases.
Note that a) will allow you to build from readonly source tree .
How old is patch ? Before generated "python solution" to be moved from
source tree to
cat pybuilddir.txt. So thanks to all that resolvedesign failure with commit for bpo-13150. It seems to me author is not
able to rewrite => bpo-13547, bpo-14774, bpo-16342(?) .
Now obsolete by bpo-15298 (10x Trent).
May be I'm wrong to use cross build for multilib , i.e. with gcc -m32.
In this case does not exist prefixed readelf.
bpo-15485
My multilib compiler list fist paths to 64 libraries the to 32. 64 bit
is with all dependent libraries and headers but 32-compatible lack some
(tk/tcl for instance). As result 32 bit build find all "required"
dependency, try to link all modules and fail on missing.
Distutil library search is limited - just search for library name.
May be (bpo-15268).
bpo-14598 and bpo-15484.
For instance is source tree is readonly . It is important in cross to
avoid build defects to update source files.
Result is parallel build and test for multiple host.
Unfortunately issue is closed. It was fixed perfectly for about 2 hours.
I just revert commit.
Same as 14 rule apply.
May be is not same as bpo-15833 (fixed+closed). May be case is only for
regression test if source tree is readonly.
Roumen
rpetrov commentedon Feb 2, 2013
I agree that cross-compilation is now usable.
Issues related to build and test outside source tree more or less are permanent but out of cross-compilation scope.
Also parts of patches posted in this issue now are in separate defects or enhancements.
GregHellings commentedon Feb 5, 2013
I'm trying to cross-compile the latest default out of Mercurial based on the status of this. Currently it fails with the following invocation:
This is in a Fedora 18 environment using its included MinGW packages to do the cross compile. I have applied no patches and just did the hg clone this morning from the official repository. Are any of these attached patches still necessary? Is there any documentation about what the process is now for doing the cross-compile build?
mingwandroid commentedon Feb 5, 2013
Yes, patches are still required.
Mainly Roumen's big patch [1] and then a load more too. Matthias Klose has merged a few cross compilation patches.
Here my project with patches for 3.3.0 and an emphasis on cross:
https://github.com/mingwandroid/crucifixion-freedom
...and here's mingwbuilds with the exact same patches (but likely tidier build scripts) without the cross focus:
https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0
Next time there's a release of Python 3, I'll rebase my patches against that.
[1] http://bugs.python.org/issue3871
GregHellings commentedon Feb 5, 2013
Bummer, the patches in that issue do not apply cleanly to either the 3.3.0 released tarball nor to the 3.3 branch from hg.
doko42 commentedon Feb 6, 2013
See http://mail.python.org/pipermail/python-dev/2013-January/123774.html
for the discussion.
Not updating the patches for tip/trunk is the best way to keep them out of the project.
packge/pkg-python: drop custom environment variables