Skip to content

cross-compilation support for python build #48004

Closed

Description

@rpetrov
mannequin
BPO 3754
Nosy @doko42, @vapier, @tarekziade, @merwok, @mingwandroid
Files
  • python-trunk-20100516-CROSS.patch
  • python-trunk-20100626-CROSS.patch
  • python-trunk-20100824-CROSS.patch
  • python-2.7-20101112-CROSS.patch
  • python-py3k-20101112-CROSS.patch
  • python-2.7-20110126-CROSS.patch
  • python-py3k-20110126-CROSS.patch
  • python-py3k-20110407-CROSS.patch
  • python-py3k-20110520-CROSS.patch
  • python-py3k-20110825-CROSS.patch
  • python-py3k-20120318-CROSS.patch
  • python-py3k-20120318-CROSS-330a2.patch: Patch updated to build against 3.3.0a2
  • python-py3k-20120607-CROSS.patch: version for 3.3.0a4
  • py3k-20121004-CROSS.tgz
  • log
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tarekziade'
    closed_at = <Date 2013-02-02.16:51:13.100>
    created_at = <Date 2008-09-01.21:49:18.597>
    labels = ['type-feature', 'build']
    title = 'cross-compilation support for python build'
    updated_at = <Date 2013-02-06.13:27:50.032>
    user = 'https://bugs.python.org/rpetrov'

    bugs.python.org fields:

    activity = <Date 2013-02-06.13:27:50.032>
    actor = 'doko'
    assignee = 'tarek'
    closed = True
    closed_date = <Date 2013-02-02.16:51:13.100>
    closer = 'rpetrov'
    components = ['Cross-Build']
    creation = <Date 2008-09-01.21:49:18.597>
    creator = 'rpetrov'
    dependencies = []
    files = ['17361', '17778', '18625', '19586', '19588', '20534', '20536', '21570', '22036', '23034', '24923', '25762', '25850', '27444', '27896']
    hgrepos = []
    issue_num = 3754
    keywords = ['patch', 'needs review']
    message_count = 56.0
    messages = ['72299', '77241', '98223', '105859', '108744', '110685', '114776', '127140', '132102', '133252', '136371', '136772', '136885', '136887', '136911', '136962', '136963', '136987', '137001', '142972', '155693', '155971', '155972', '155977', '155980', '156240', '161972', '162421', '162464', '164376', '164378', '164381', '164382', '164385', '164395', '164396', '164402', '164403', '164409', '164412', '164416', '166740', '171787', '172156', '174925', '174926', '174928', '175009', '180590', '180594', '180701', '181180', '181447', '181449', '181457', '181520']
    nosy_count = 18.0
    nosy_names = ['doko', 'wrobell', 'vapier', 'tarek', 'eric.araujo', 'rpetrov', 'Arfrever', 'devurandom', 'WhiteTiger', 'BreamoreBoy', 'python-dev', 'David.Kern', 'Greg.Hellings', 'kalev', 'marqvar', 'Ray.Donnelly', 'eudoxos', 'ambrop7']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue3754'
    versions = ['Python 3.3']

    Activity

    1. wrobell commented on May 25, 2011

      wrobellmannequin
      Mannequin

      Would it be possible to get list of steps required to test this patch?

      1. hg clone
      2. copy config.sub and config.guess?
      3. autoreconf
      4. configure...
      5. make

      What else?

    2. rpetrov commented on May 25, 2011

      rpetrovmannequin
      MannequinAuthor

      Usually this is not a question for bug-tracking system . configure ..
      --host=.... will enter into cross-compilation mode. You could find more
      details into respective manual.

      Roumen

    3. wrobell commented on May 25, 2011

      wrobellmannequin
      Mannequin

      well, looking at the first comment, there is required more than simple read of the manual (i.e. missing config.guess/config.sub).

      is anything else missing or required?

    4. wrobell commented on May 26, 2011

      wrobellmannequin
      Mannequin

      as well, by default one experiences

      checking for /dev/ptmx... not set
      configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

      the workaround:

      1. create config.site file with the following lines
           ac_cv_file__dev_ptmx=no
           ac_cv_file__dev_ptc=no
      1. run configure script as: CONFIG_SITE=config.site ./configure --host...

      would be it ok if i add appropriate options for /dev/ptmx and /dev/ptc to this patch so we do not have to play with config.site file?

    5. wrobell commented on May 26, 2011

      wrobellmannequin
      Mannequin

      next question. when starting compilation i am getting

      In file included from Include/Python.h:50,
      from ./Modules/python.c:3:
      Include/pyport.h:257:13: error: #error "This platform's pyconfig.h needs to define PY_FORMAT_LONG_LONG"
      make: *** [Modules/python.o] Error 1

      how to deal with above?

    6. wrobell commented on May 26, 2011

      wrobellmannequin
      Mannequin

      looking into configure.in the above fails due to following check

      AC_MSG_CHECKING(for %lld and %llu printf() format support)

      the check compiles and tries to _run_ a bit of software to determine lld/llu support.

      that of course fails (we are cross compiling). this is similar problem to ptmx/ptc problem (we do not know if host have support for ptmx/ptc).

      i wonder if the following would be acceptable

      1. if in cross compilation mode, the configure.in could load config-cross.site.
      2. the config-cross.site would define variables for ptmx/ptc, long long format and other checks which consist of running code snippets
      3. appropriate checks (i.e. ptmx/ptc, lld/llu) would not be run in cross compilation mode. variables from config-cross.site would be used instead.
      4. it would be a developer responsibility to set appropriate values for his platform in config-cross.site file.

      does above sound like a solution?

    7. rpetrov commented on May 26, 2011

      rpetrovmannequin
      MannequinAuthor

      At least one is really would like to cross-compile.

      worber, the config site has to look like (sample for linux i?86, i.e. intel 32 bit, as host platform)
      =================================

      ac_cv_little_endian_double=yes
      ac_cv_broken_sem_getvalue=no
      ac_cv_computed_gotos=yes
      ac_cv_buggy_getaddrinfo=no
      ac_cv_working_tzset=yes
      #next line require fix typo in configure.in
      ac_cv_pthread=yes
      ac_cv_pthread_system_supported=yes
      ac_cv_tanh_preserves_zero_sign=yes
      
      ac_cv_have_long_long_format=yes
      ac_cv_file__dev_ptmx=yes
      ac_cv_file__dev_ptc=no

      =================================

      Also with following patch into configure.in
      =======================================

      diff --git a/configure.in b/configure.in
      --- a/configure.in
      +++ b/configure.in
      @@ -1372,7 +1372,7 @@
       # so we need to run a program to see whether it really made the
       # function available.
       AC_MSG_CHECKING(whether $CC accepts -pthread)
      -AC_CACHE_VAL(ac_cv_thread,
      +AC_CACHE_VAL(ac_cv_pthread,
       [ac_save_cc="$CC"
       CC="$CC -pthread"
       AC_RUN_IFELSE([AC_LANG_SOURCE([[

      =======================================

    8. merwok commented on Aug 25, 2011

      @merwok
      Member

      Copy-pasting of a previous comment: I see some “fix for issue #NNN is bogus” in your patch: would you open separate bug reports for those? A diff file is not a very useful way to report bugs or express opinions <wink>.

      Copy-pasting of many previous comments: distutils does not get new features (cross-compilation is a new feature), please focus on packaging.

    9. doko42 commented on Mar 14, 2012

      @doko42
      Member

      Roumen, do you have a newer patch which applies to the current trunk?

    10. doko42 commented on Mar 15, 2012

      @doko42
      Member

      some of these changes are now available in trunk, tested with a x86_64-linux-gnu to arm-linux-gnueabihf cross build.

      using bpo-14324 for configure tests which need fixes, and bpo-14327 for the use of uname in the configure script.

      please feel free to open separate issues for the changes in the distutils package and the packaging package.

    11. merwok commented on Mar 15, 2012

      @merwok
      Member

      Not for distutils (feature-frozen, see previous messages)

    12. doko42 commented on Mar 16, 2012

      @doko42
      Member

      bpo-14330 handles not using the host python for the build

    13. doko42 commented on Mar 16, 2012

      @doko42
      Member

      I'm now able to build the interpreter (and run it in qemu on the same machine), but building the extension modules fails. the build is configured as:

      CC="arm-linux-gnueabihf-gcc" \
      CXX="arm-linux-gnueabihf-g++" \
      CFLAGS="-g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security" \
      LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" \
          ../cross/configure \
              --prefix=/usr \
              --enable-ipv6 \
              --with-computed-gotos \
              --enable-loadable-sqlite-extensions \
              --with-dbmliborder=bdb:gdbm \
              --with-system-expat \
              --with-system-ffi \
              --with-fpectl \
              --host=arm-linux-gnueabihf \
              --build=x86_64-linux-gnu

      building 'cmath' extension
      arm-linux-gnueabihf-gcc -fPIC -Wno-unused-result -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -I. -I/usr/include/arm-linux-gnueabihf -I/usr/local/include -I/usr/include/python3.3m -c cmathmodule.c -o build/temp.linux-x86_64-3.3/cmathmodule.o
      arm-linux-gnueabihf-gcc: error: cmathmodule.c: No such file or directory
      arm-linux-gnueabihf-gcc: fatal error: no input files
      compilation terminated.

      the source files are not found (srcdir != builddir), the patch from rpetrov talks about:

      +# revert patch from bpo-7880 :
      +# - the test case (from bpo-7880) works for me
      +# - realpath break cross compilation
      +# => so lets use abspath again ;)
      +# NOTE "Issue bpo-6612: Fix site and sysconfig to catch os.getcwd() error,
      +# eg. if the current directory was deleted." replase all occurrence of
      +# realpath with _safe_realpath. Instead to update _safe_realpath to use
      +# abspath the cross-compilation revert all!

      but I'm running out of time, and won't work on this for the next two weeks.

    14. rpetrov commented on Mar 18, 2012

      rpetrovmannequin
      MannequinAuthor

      I cannot test arm build due to bpo-12010

    15. mingwandroid commented on May 30, 2012

      mingwandroidmannequin
      Mannequin

      Hi Roumen,

      Many thanks for your patches, I've been using a 2.7.1 version of your patches for Python integration with GDB (pretty-printing) of my own version of the Android NDK for ages now (part of the Necessitas Qt project) and I really appreciate your efforts in this area. I've recently started looking into cross compilation of Python as I want to build everything on Linux and hopefully I can be useful in this effort.

      What you said about wchar_t on Android is true on API level <= 8, but fixed for >8. See:

      https://groups.google.com/group/android-ndk/browse_thread/thread/e40e3fd241e72d3f?pli=1

      To test this:

      test.c:
      #include <stdlib.h>
      char bytes[(signed)sizeof(wchar_t)-4];

      compile failure:
      arm-linux-androideabi-gcc -I$NDK/platforms/android-8/arch-arm/usr/include/ -S test.c
      test.c:2: error: size of array 'bytes' is negative

      compile success:
      arm-linux-androideabi-gcc -I$NDK/platforms/android-9/arch-arm/usr/include/ -S test.c

      I've attached a version of your latest patch re-made against released 3.3.0a2. I've also done the same thing for your MinGW patch at http://bugs.python.org/issue3871

      I'll follow up shortly with a patch that addresses some of the remaining issues with cross building for MinGW on Linux (and also the beginnings of Darwin on Linux). It'll be attached to http://bugs.python.org/issue3871 though as it's more specific to MinGW than general cross.

    16. rpetrov commented on Jun 6, 2012

      rpetrovmannequin
      MannequinAuthor

      About Android issue with wchar still is applicable. So issue is not exactly as in 12010. Even with changed size multi-byte functions just return return result like a byte array, i.e. without conversion. Since python try to validate wide character at some point will be raised exception invalid multi-byte.
      Only for test work around could be implement - "fake" conversion functions that just store char into wchar array and to use ascii codec.
      Next is that lack of locale support and lack of locale information(structure is without members).
      Another issue is that loader does not resolve indirect dependencies. Many python modules depend from math library but are not linked with it and math functions are resolved later. This does not work on android and those modules must be explicitly linked to math library.
      Also posix semaphores must be disabled.

      As result python 3k regression tests except those that require locale support or ctype will pass.

    17. mingwandroid commented on Jun 7, 2012

      mingwandroidmannequin
      Mannequin

      Re Android

      Ah of course. In that case, there's always the NDK built by Dmitry (his latest is v7 AFAIK):

      http://www.crystax.net/en/android/ndk

    18. python-dev commented on Jun 30, 2012

      python-devmannequin
      Mannequin

      New changeset 56c574a1819a by doko in branch 'default':

    19. python-dev commented on Jun 30, 2012

      python-devmannequin
      Mannequin

      New changeset af15910d2ffb by doko in branch 'default':

    20. doko42 commented on Jun 30, 2012

      @doko42
      Member

      updated the patch in issue bpo-14330.

    21. doko42 commented on Jun 30, 2012

      @doko42
      Member

      some chunks of the python-py3k-20120607-CROSS.patch patch are now checked in.

      I didn't see any issues with the symlinks, and generating the posix vars, so maybe these bits should be dropped from the patch.

      remaining issues are:

      • the readline/ncurses configury. maybe this should be done
        in the configure.ac using the host tools, and then provided
        to setup.py by a Makefile macro?

      • why are the Makefile changes for setup.info, pyconfig.h needed?

      • is the _testembed change still needed?

      • does Python/importlib.h really need a rebuild if it's there?
        I think this should just fail, not removed for the cross build.

      • why is the ac_cv_thread vs. ac_cv_pthread change needed?

    22. python-dev commented on Jun 30, 2012

      python-devmannequin
      Mannequin

      New changeset d158b0a78390 by doko in branch 'default':

    23. doko42 commented on Jun 30, 2012

      @doko42
      Member

      for the readline ldd check, I'm checking in a patch to use readelf instead of ldd for the cross build.

    24. python-dev commented on Jun 30, 2012

      python-devmannequin
      Mannequin

      New changeset e6e99d449bdc by doko in branch 'default':

    25. doko42 commented on Jun 30, 2012

      @doko42
      Member

      the ncurses/_flags changes seem to be unrelated. please open a separate issue.

    26. python-dev commented on Jun 30, 2012

      python-devmannequin
      Mannequin

      New changeset 12a56a349af2 by doko in branch 'default':

    27. mingwandroid commented on Jun 30, 2012

      mingwandroidmannequin
      Mannequin

      NCURSES_INTERNALS stuff appears to be redundant:

      Mac OS X curses.h, Linux curses.h and Windows PDCurses.h don't reference it, nor does the Python 3.3.0b1 source code. Of course, I haven't checked any other systems.

      However: see http://bugs.python.org/issue14598, so maybe for older Linux ncurses and Cygwin it is still needed.

      NCURSES_OPAQUE investigation:

      Linux ncurses.h:
      #ifndef NCURSES_OPAQUE
      #define NCURSES_OPAQUE 0
      #endif
      
      Mac OS X ncurses.h:
      #ifndef NCURSES_OPAQUE
      #define NCURSES_OPAQUE 1
      #endif

      So for OSX we need to be defining it to 0...

      but IMHO, these parts are not directly related to cross compilation support so shouldn't be part of this patch.

    28. doko42 commented on Jun 30, 2012

      @doko42
      Member

      Roumen, I would like to close this issue. Please could you file separate issues for the remaining bits?

      • the thread/pthread configure issue

      • the generation of the Setup / pyconfig.h files?

    29. rpetrov commented on Jun 30, 2012

      rpetrovmannequin
      MannequinAuthor

      Matthias I cannot follow all you questions as I'm on vacation so briefly:
      a) ac_cv_thread is type correct one is with'p'
      b) ncurses
      NCURSES_INTERNALS reported again in bpo-14598
      NCURSES_OPAQUE - i don't have time to explain all details but it could be reproduces on linux with version 2.7
      Even on linux to linux build it is required as it could depend from ncurses library version.

      c) setup_info is my proposition to pass detection of readline libs from configure to setup.py . Use of readelf is not portable. Why to test again ? If on host platforms shared library is not in elf format which tool to use ?

      d) importlib.h is build now from executable. Probably we could introduce BUILD_CC . Out of scope for now

      e) executable that load packages has to be located in builddir to allow "is python build" to be activated and packages from cross-build environment to be loaded.

      After 3 weeks I could answer more.

    30. rpetrov commented on Jul 29, 2012

      rpetrovmannequin
      MannequinAuthor

      I hope that following separate issues will address remaining part of this patch:

      required:

      • 15483: initialise include and library paths in setup.py
      • 15484: use _PYTHON_PROJECT_BASE in distutils sysconfig
      • TODO: typo in in configure.in - use ac_cv_pthread instead ac_cv_thread
      • 15268: skip "curses configure checks fail if only /usr/include/ncursesw/curses.h is installed"

      optional:

      • 15298: _sysconfigdata is generated in srcdir, not builddir
      • 14598: _cursesmodule.c fails with ncurses-5.9 on Linux reported first as 14438 - _cursesmodule build fails on cygwin
        and all other that allow python to be build smoothly outside source directory
    31. added
      buildThe build process and cross-build
      and removed
      buildThe build process and cross-build
      on Jul 29, 2012
    32. eudoxos commented on Oct 2, 2012

      eudoxosmannequin
      Mannequin

      Being a newcomer to this issue, I would like to ask for a brief summary about which parts of the patch are checked in for 3.3.0 and which are still to be applied.

      Roumen mentions bpo-15483, bpo-15484, bpo-15268 and the ac_cv_thread in the previous post as mandatory and bpo-15298 and bpo-14598 as optional; can I add those as dependencies of this issue?

      python-py3k-20120607-CROSS.patch does not apply to 3.3.0 cleanly, but looking quickly at .rej files suggests that rejected hunks are those which were already merged for 3.3.0 final. Can someone confirm that?

    33. rpetrov commented on Oct 6, 2012

      rpetrovmannequin
      MannequinAuthor

      Hi Václav,
      uploaded file py3k-20121004-CROSS.tgz contain current status of patch extracted as is from my repository, i.e. on small chunks. I have no idea what is in 3.3.0 .

    34. ambrop7 commented on Nov 5, 2012

      ambrop7mannequin
      Mannequin

      I've applied all patches from py3k-20121004-CROSS.tgz to Python 3.3.0 except 2, 4, and 9 (which didn't apply), but it's not working. After it builds the built in modules, it tries to run the parser generator which was cross-compiled.

      make Parser/pgen
      make[1]: Entering directory /root/Python-3.3.0' armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/dynamic_annotations.o](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.o) [Python/dynamic_annotations.c](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/mysnprintf.o](https://github.com/python/cpython/blob/main/Python/mysnprintf.o) [Python/mysnprintf.c](https://github.com/python/cpython/blob/main/Python/mysnprintf.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Python/pyctype.o](https://github.com/python/cpython/blob/main/Python/pyctype.o) [Python/pyctype.c](https://github.com/python/cpython/blob/main/Python/pyctype.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/tokenizer_pgen.o](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.o) [Parser/tokenizer_pgen.c](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/printgrammar.o](https://github.com/python/cpython/blob/main/Parser/printgrammar.o) [Parser/printgrammar.c](https://github.com/python/cpython/blob/main/Parser/printgrammar.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/parsetok_pgen.o](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.o) [Parser/parsetok_pgen.c](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.c) armv5tel-softfloat-linux-gnueabi-gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -o [Parser/pgenmain.o](https://github.com/python/cpython/blob/main/Parser/pgenmain.o) [Parser/pgenmain.c](https://github.com/python/cpython/blob/main/Parser/pgenmain.c) armv5tel-softfloat-linux-gnueabi-gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes [Parser/acceler.o](https://github.com/python/cpython/blob/main/Parser/acceler.o) [Parser/grammar1.o](https://github.com/python/cpython/blob/main/Parser/grammar1.o) [Parser/listnode.o](https://github.com/python/cpython/blob/main/Parser/listnode.o) [Parser/node.o](https://github.com/python/cpython/blob/main/Parser/node.o) [Parser/parser.o](https://github.com/python/cpython/blob/main/Parser/parser.o) [Parser/bitset.o](https://github.com/python/cpython/blob/main/Parser/bitset.o) [Parser/metagrammar.o](https://github.com/python/cpython/blob/main/Parser/metagrammar.o) [Parser/firstsets.o](https://github.com/python/cpython/blob/main/Parser/firstsets.o) [Parser/grammar.o](https://github.com/python/cpython/blob/main/Parser/grammar.o) [Parser/pgen.o](https://github.com/python/cpython/blob/main/Parser/pgen.o) [Objects/obmalloc.o](https://github.com/python/cpython/blob/main/Objects/obmalloc.o) [Python/dynamic_annotations.o](https://github.com/python/cpython/blob/main/Python/dynamic_annotations.o) [Python/mysnprintf.o](https://github.com/python/cpython/blob/main/Python/mysnprintf.o) [Python/pyctype.o](https://github.com/python/cpython/blob/main/Python/pyctype.o) [Parser/tokenizer_pgen.o](https://github.com/python/cpython/blob/main/Parser/tokenizer_pgen.o) [Parser/printgrammar.o](https://github.com/python/cpython/blob/main/Parser/printgrammar.o) [Parser/parsetok_pgen.o](https://github.com/python/cpython/blob/main/Parser/parsetok_pgen.o) [Parser/pgenmain.o](https://github.com/python/cpython/blob/main/Parser/pgenmain.o) -lpthread -ldl -lpthread -lutil -o [Parser/pgen](https://github.com/python/cpython/blob/main/Parser/pgen) make[1]: Leaving directory /root/Python-3.3.0'
      Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
      Parser/pgen: Parser/pgen: cannot execute binary file
      make: *** [Include/graminit.h] Error 126

    35. ambrop7 commented on Nov 5, 2012

      ambrop7mannequin
      Mannequin

      Forgot to mention: I did run autoreconf after applying the patches. I'm attaching the full output.

    36. ambrop7 commented on Nov 5, 2012

      ambrop7mannequin
      Mannequin

      A minor issue: if only --host= is specified on command line but not --build=, then cross_compiling variable is only defined after AC_PROG_CC is called. However, configure.ac uses it before that (e.g. the part at the top which looks for a python interpreter). Well, more precisely, something in autoconf sets it to cross_compiling=maybe.

    37. rpetrov commented on Nov 6, 2012

      rpetrovmannequin
      MannequinAuthor

      HI, Ambroz

      Ambroz Bizjak wrote:

      I've applied all patches from py3k-20121004-CROSS.tgz to Python 3.3.0 except 2, 4, and 9 (which didn't apply), but it's not working. After it builds the built in modules, it tries to run the parser generator which was cross-compiled.
      3.3.0 was split from head long time before release and I did not spend
      time to check what is actually entered.

      About patches :
      2: restore graminit.* to source directory
      4: restore AST to source directory

      After this touch files to avoid timestamp dependencies to launch extra
      build .
      You issue was reported and I guess is documented . It is related to
      checkout from repository and to general build process.
      I have a shell script to this but in brief script touch in source tree
      following files:

      If you decide to build in source tree patch 2 and 4 could be skipped .
      Note that listed patches allow user to keep separate builds at same time
      for different platforms. As side effect part of patches could help to
      build with different configuration options.

      For one time cross-build users could use following scenario:

      1. native configuration, build and installation
      2. clean source tree
      3. cross build&pack for one platform.
        Repeat 2 & 3 for each platform

      about 9: pass all top configure arguments to libffi configure to allow
      users to build it properly
      It is save to ignore this patch if libffi build is fine for you host
      platform. Otherwise .... just change setup to use only
      ...
      config_args = sysconfig.get_config_var("CONFIG_ARGS")
      ...
      and replace " ".join(config_args) with config_args in command.

      For instance I have build for a platform and with filtered "configure
      arguments" build fail. Instead to patch libffi code I prefer from main
      configure command line to control build of libffi.

      Roumen

    38. python-dev commented on Jan 25, 2013

      python-devmannequin
      Mannequin

      New changeset 11a18263ceb7 by doko in branch '2.7':

      New changeset e28b30e6eee6 by doko in branch '3.2':

      New changeset 5464a534e7bd by doko in branch '3.3':

      New changeset ee48728e3695 by doko in branch 'default':

    39. doko42 commented on Jan 25, 2013

      @doko42
      Member

      about py3k-20121004-CROSS.tgz:

      • committed 0001-CROSS-fix-typo-in-thread-AC_CACHE_VAL.patch

      • 0002-CROSS-restore-graminit.-to-source-directory.patch
        is this necessary? Assuming that you have correct time
        stamps, this is something which usually is not built during
        a cross build.

        Can we just have a shell script which touches some files
        so that you can cross-build even from the repository without
        building pgen?

        Do the release managers have such a script which is run
        when building a release?

      • 0003-CROSS-restore-importlib-header-to-source-directory-a.patch

        Same as for 0002, using correct timestamps. The EXEEXT is a
        different issue. However I'm not sure what to use, EXEEXT,
        or BUILDEXEEXT.

      • 0004-CROSS-restore-AST-to-source-directory.patch

        Same as for 0002. Btw, does this fail with a native out-of-tree
        build too?

      • 0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch
        A comment like "The fix for issue bpo-13150 is bogus and break
        everything" might not the best way to convince maintainers ...

        This works for me, and IMO should be dropped, or please explain
        why this doesn't work for you.

      • 0006-CROSS-initialise-include-and-library-paths.patch
        Afaics, a similar patch is now applied. Can be dropped.

      • 0007-CROSS-set-_PYTHON_PROJECT_BASE-to-current-build-dir.patch
        Applied.

      • 0008-CROSS-use-_PYTHON_PROJECT_BASE-in-distutils-sysconfi.patch
        Applied.

      • 0009-CROSS-pass-all-top-configure-arguments-to-libffi-con.patch

        Why is it needed to pass all configure args?

      • 0010-CROSS-warn-only-if-readelf-is-not-in-host-triplet-fo.patch

        Why is this needed just for readelf? If you do have a cross
        toolchain installed, then this should be available with
        the triplet name, same as for gcc, as, ...

      • 0011-CROSS-append-gcc-library-search-paths-instead-to-pre.patch

        Looks ok. Could you mention where this does make a difference,
        or do you just want to keep the search order as defined by
        the compiler?

      • 0012-CROSS-avoid-ncursesw-include-path-hack.patch

        This would break cross-building in a multiarch environment.
        Maybe we need a --with-curses-incdir=<> option.

      • 0013-CROSS-properly-detect-WINDOW-_flags-for-different-nc.patch
        This should be a separate issue. Not cross specific.

      • 0014-CROSS-use-build-directory-as-root-for-regression-tes.patch
        why is this necessary?

      • 0015-CROSS-test-tools-has-to-depend-only-from-location-of.patch
        Looks ok, but not cross specific. Could you file a separate issue?

      • 0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch
        Looks like a separate issue.

    40. rpetrov commented on Jan 26, 2013

      rpetrovmannequin
      MannequinAuthor

      Matthias Klose wrote:

      Matthias Klose added the comment:
      about py3k-20121004-CROSS.tgz:
      [SNIP]

      • 0002-CROSS-restore-graminit.-to-source-directory.patch
        [SNIP]
      • 0003-CROSS-restore-importlib-header-to-source-directory-a.patch
        [SNIP]
      • 0004-CROSS-restore-AST-to-source-directory.patch
        [SNIP]

      All above is related to build out of source tree.
      Try to keep a build tree for long time. Synchronize source, for instance
      once in week, and try to rebuild. After updates in those files python
      (native build) crash.
      Solutions:
      a) restore to source tree as before and only for cross build touch files
      to avoid regeneration.
      b) after synchronization copy to build tree but now in all cases.
      Note that a) will allow you to build from readonly source tree .

      • 0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch
        A comment like "The fix for issue bpo-13150 is bogus and break
        everything" might not the best way to convince maintainers ...

        This works for me, and IMO should be dropped, or please explain
        why this doesn't work for you.

      How old is patch ? Before generated "python solution" to be moved from
      source tree to cat pybuilddir.txt. So thanks to all that resolve
      design failure with commit for bpo-13150. It seems to me author is not
      able to rewrite => bpo-13547, bpo-14774, bpo-16342(?) .
      Now obsolete by bpo-15298 (10x Trent).

      • 0006-CROSS-initialise-include-and-library-paths.patch
        Afaics, a similar patch is now applied. Can be dropped.

      [SNIP]

      • 0009-CROSS-pass-all-top-configure-arguments-to-libffi-con.patch

        Why is it needed to pass all configure args?

      • 0010-CROSS-warn-only-if-readelf-is-not-in-host-triplet-fo.patch

        Why is this needed just for readelf? If you do have a cross
        toolchain installed, then this should be available with
        the triplet name, same as for gcc, as, ...

      May be I'm wrong to use cross build for multilib , i.e. with gcc -m32.
      In this case does not exist prefixed readelf.

      • 0011-CROSS-append-gcc-library-search-paths-instead-to-pre.patch

        Looks ok. Could you mention where this does make a difference,
        or do you just want to keep the search order as defined by
        the compiler?

      bpo-15485
      My multilib compiler list fist paths to 64 libraries the to 32. 64 bit
      is with all dependent libraries and headers but 32-compatible lack some
      (tk/tcl for instance). As result 32 bit build find all "required"
      dependency, try to link all modules and fail on missing.
      Distutil library search is limited - just search for library name.

      • 0012-CROSS-avoid-ncursesw-include-path-hack.patch

        This would break cross-building in a multiarch environment.
        Maybe we need a --with-curses-incdir=<> option.

      May be (bpo-15268).

      • 0013-CROSS-properly-detect-WINDOW-_flags-for-different-nc.patch
        This should be a separate issue. Not cross specific.

      bpo-14598 and bpo-15484.

      • 0014-CROSS-use-build-directory-as-root-for-regression-tes.patch
        why is this necessary?

      For instance is source tree is readonly . It is important in cross to
      avoid build defects to update source files.
      Result is parallel build and test for multiple host.

      • 0015-CROSS-test-tools-has-to-depend-only-from-location-of.patch
        Looks ok, but not cross specific. Could you file a separate issue?

      Unfortunately issue is closed. It was fixed perfectly for about 2 hours.
      I just revert commit.
      Same as 14 rule apply.

      • 0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch
        Looks like a separate issue.

      May be is not same as bpo-15833 (fixed+closed). May be case is only for
      regression test if source tree is readonly.

      Roumen

    41. rpetrov commented on Feb 2, 2013

      rpetrovmannequin
      MannequinAuthor

      I agree that cross-compilation is now usable.
      Issues related to build and test outside source tree more or less are permanent but out of cross-compilation scope.
      Also parts of patches posted in this issue now are in separate defects or enhancements.

    42. GregHellings commented on Feb 5, 2013

      GregHellingsmannequin
      Mannequin

      I'm trying to cross-compile the latest default out of Mercurial based on the status of this. Currently it fails with the following invocation:

      $ ../configure --host=i686-w64-mingw32 --build=x86_64-redhat-linux-gnu --target=i686-w64-mingw32 --prefix=/usr/i686-w64-mingw32/sys-root/mingw --exec-prefix=/usr/i686-w64-mingw32/sys-root/mingw --bindir=/usr/i686-w64-mingw32/sys-root/mingw/bin --sbindir=/usr/i686-w64-mingw32/sys-root/mingw/sbin --sysconfdir=/usr/i686-w64-mingw32/sys-root/mingw/etc --datadir=/usr/i686-w64-mingw32/sys-root/mingw/share --includedir=/usr/i686-w64-mingw32/sys-root/mingw/include --libdir=/usr/i686-w64-mingw32/sys-root/mingw/lib --libexecdir=/usr/i686-w64-mingw32/sys-root/mingw/libexec --localstatedir=/usr/i686-w64-mingw32/sys-root/mingw/var --sharedstatedir=/usr/i686-w64-mingw32/sys-root/mingw/com --mandir=/usr/i686-w64-mingw32/sys-root/mingw/share/man --infodir=/usr/i686-w64-mingw32/sys-root/mingw/share/info --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --with-system-expat --with-system-ffi --with-pydebug --with-tsc
      checking for hg... found
      checking build system type... x86_64-redhat-linux-gnu
      checking host system type... i686-w64-mingw32
      checking for python interpreter for cross build... python3
      checking for --enable-universalsdk... no
      checking for --with-universal-archs... 32-bit
      checking MACHDEP... configure: error: cross build not supported for i686-w64-mingw32
      error: Bad exit status from /var/tmp/rpm-tmp.sZjjCl (%build)

      This is in a Fedora 18 environment using its included MinGW packages to do the cross compile. I have applied no patches and just did the hg clone this morning from the official repository. Are any of these attached patches still necessary? Is there any documentation about what the process is now for doing the cross-compile build?

    43. mingwandroid commented on Feb 5, 2013

      mingwandroidmannequin
      Mannequin

      Yes, patches are still required.

      Mainly Roumen's big patch [1] and then a load more too. Matthias Klose has merged a few cross compilation patches.

      Here my project with patches for 3.3.0 and an emphasis on cross:
      https://github.com/mingwandroid/crucifixion-freedom

      ...and here's mingwbuilds with the exact same patches (but likely tidier build scripts) without the cross focus:
      https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0

      Next time there's a release of Python 3, I'll rebase my patches against that.

      [1] http://bugs.python.org/issue3871

    44. GregHellings commented on Feb 5, 2013

      GregHellingsmannequin
      Mannequin

      Bummer, the patches in that issue do not apply cleanly to either the 3.3.0 released tarball nor to the 3.3 branch from hg.

    45. doko42 commented on Feb 6, 2013

      @doko42
      Member

      See http://mail.python.org/pipermail/python-dev/2013-January/123774.html
      for the discussion.

      Not updating the patches for tip/trunk is the best way to keep them out of the project.

    46. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    buildThe build process and cross-buildtype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions