Skip to content

setup.py: do not add system header locations when cross compiling #64410

Closed as not planned
Closed as not planned
setup.py: do not add system header locations when cross compiling#64410
Labels
3.7 (EOL)end of lifebuildThe build process and cross-build

Description

@tpetazzoni
tpetazzoni
mannequin
opened on Jan 9, 2014
BPO 20211
Nosy @doko42, @vstinner, @zware, @tpetazzoni, @yan12125
Files
  • 0001-setup.py-do-not-add-invalid-header-locations.patch
  • no-native-headers-libraries.patch
  • multiarch.patch
  • multiarch_2.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2014-01-09.23:02:08.889>
labels = ['build', '3.7']
title = 'setup.py: do not add system header locations when cross compiling'
updated_at = <Date 2019-12-10.08:12:27.238>
user = 'https://github.com/tpetazzoni'

    bugs.python.org fields:

    activity = <Date 2019-12-10.08:12:27.238>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Cross-Build']
creation = <Date 2014-01-09.23:02:08.889>
creator = 'thomas-petazzoni'
dependencies = []
files = ['33393', '45830', '45893', '45894']
hgrepos = []
issue_num = 20211
keywords = ['patch']
message_count = 12.0
messages = ['207805', '237380', '261435', '282640', '282641', '282836', '282837', '282841', '282843', '283182', '283183', '283184']
nosy_count = 7.0
nosy_names = ['doko', 'vstinner', 'python-dev', 'zach.ware', 'thomas-petazzoni', 'yan12125', 'hundeboll']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue20211'
versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. tpetazzoni commented on Jan 9, 2014

      @tpetazzoni
      tpetazzonimannequin
      on Jan 9, 2014
      MannequinAuthor

      In the cross-compilation case, setup.py incorrectly adds /usr/include to self.compiler.include_dirs, and results in the following invalid compilation line:

      /home/thomas/projets/buildroot/output/host/usr/bin/arm-none-linux-gnueabi-gcc
      -fPIC -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -g
      -O3 -Wall -Wstrict-prototypes -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
      -D_FILE_OFFSET_BITS=64 -pipe -Os
      -I./Include -I/usr/include -I. -IInclude
      -I/home/thomas/projets/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/include
      -I/home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Include
      -I/home/thomas/projets/buildroot/output/build/python3-3.4.0b1
      -c /home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Modules/_struct.c
      -o build/temp.linux-arm-3.4/home/thomas/projets/buildroot/output/build/python3-3.4.0b1/Modules/_struct.o
      cc1: warning: include location "/usr/include" is unsafe for cross-compilation [-Wpoison-system-directories]

      The -I/usr/include is wrong when cross compiling, so we disable adding INCLUDEDIR and LIBDIR from the host when cross compiling.

    2. tpetazzoni
      added
      buildThe build process and cross-build
      on Jan 9, 2014

    3. BreamoreBoy commented on Mar 6, 2015

      @BreamoreBoy
      BreamoreBoymannequin
      on Mar 6, 2015
      Mannequin

      Can we have a patch review on this issue please.

    4. hundeboll commented on Mar 9, 2016

      @hundeboll
      hundebollmannequin
      on Mar 9, 2016
      Mannequin

      The patch looks good to me, and works in my setup.

    5. yan12125 commented on Dec 7, 2016

      @yan12125
      yan12125mannequin
      on Dec 7, 2016
      Mannequin

      Got the same issue here on building Python for Android. See https://circleci.com/gh/yan12125/python3-android/11 for an example build log. /usr/include is erroneously included and leads to build failures:

      building 'xxlimited' extension
/home/ubuntu/android-ndk-r13/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target aarch64-none-linux-android -gcc-toolchain /home/ubuntu/android-ndk-r13/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/home/ubuntu/android-ndk-r13/platforms/android-21/arch-arm64/usr -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIE -fno-integrated-as -fPIE -fno-integrated-as -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -DPy_LIMITED_API=0x03050000 -I../Include -I/usr/include -IObjects -IPython -I. -I/home/ubuntu/python3-android/build/21-aarch64-linux-android-4.9/usr/include -I/home/ubuntu/python3-android/src/cpython/Include -I/home/ubuntu/python3-android/src/cpython/build-target -c /home/ubuntu/python3-android/src/cpython/Modules/xxlimited.c -o build/temp.linux-aarch64-3.7/home/ubuntu/python3-android/src/cpython/Modules/xxlimited.o
In file included from /home/ubuntu/python3-android/src/cpython/Modules/xxlimited.c:17:
In file included from ../Include/Python.h:11:
In file included from /usr/include/limits.h:25:
/usr/include/features.h:398:10: fatal error: 'gnu/stubs.h' file not found
#include <gnu/stubs.h>
         ^
1 error generated.

      The patch looks reasonable and still applies to the default branch. With this patch the build is fine: https://circleci.com/gh/yan12125/python3-android/12

    6. yan12125 commented on Dec 7, 2016

      @yan12125
      yan12125mannequin
      on Dec 7, 2016
      Mannequin

      Modified this issue a bit and add some experts on the build system

    7. yan12125
      added
      3.7 (EOL)end of life
      on Dec 7, 2016
    8. yan12125
      changed the title [-]setup.py: do not add invalid header locations[/-] [+]setup.py: do not add system header locations when cross compiling[/+] on Dec 7, 2016

    9. xdegaye commented on Dec 10, 2016

      @xdegaye
      xdegayemannequin
      on Dec 10, 2016
      Mannequin

      When cross compiling, setup.py must not look for headers in the directory for installing C header files and must not look for libraries in the directory for installing object code libraries of the _native_ interpreter which is being used to run setup.py.
      Thomas patch updated to the tip of the default branch.

    10. xdegaye
      self-assigned this
      on Dec 10, 2016

    11. doko42 commented on Dec 10, 2016

      @doko42
      doko42
      on Dec 10, 2016
      Member

      this assumption is wrong for the multiarch case.

    12. yan12125 commented on Dec 10, 2016

      @yan12125
      yan12125mannequin
      on Dec 10, 2016
      Mannequin

      xdegaye's explanation is incorrect. The actual scene is much more complicated. First, let me explain why things may be broken. The wrong directory is included only if all of the following conditions are met:

      1. PYTHON_FOR_BUILD is not installed with --prefix=/usr. A common example is those which are installed with pyenv. This condition is here simply because of os.path.normpath(sys.base_prefix) != '/usr'
      2. The build script uses $DESTDIR convention to compile CPython
      3. Header files in $DESTDIR$PREFIX/include are not equivalent to $PREFIX/include. Here "equivalent" means they are for the same set of architecture/kernel/... This is a common case when cross compiling, while a famous exception is the multiarch framework on Debian-based systems.

      For example, if I use --prefix=/usr and DESTDIR=/tmp/python3-android/build, INCLUDEDIR will be /usr/include, which should not be included. On the other hand, if I use --prefix=/tmp/python3-android/build/usr and omit DESTDIR, INCLUDEDIR is correct /tmp/python3-android/build/usr/include. With $DESTDIR specified, the build script has the responsibility to specify $DESTDIR$PREFIX/include in CPPFLAGS, or CPython interpreter core won't even build, so skipping adding INCLUDEDIR in setup.py is safe. However, for the latter case (not using DESTDIR), the build script still needs to specify correct CPPFLAGS. I see it as a CPython limitation and worth documentation.

      For Debian's multiarch framework, paths are already handled by add_multiarch_paths(). I'm not sure whether $PREFIX/include should still be included or not. (I don't use Debian-based systems)

    13. xdegaye commented on Dec 10, 2016

      @xdegaye
      xdegayemannequin
      on Dec 10, 2016
      Mannequin

      this assumption is wrong for the multiarch case.

      Please explain why it is wrong.

    14. python-dev commented on Dec 14, 2016

      @python-dev
      python-devmannequin
      on Dec 14, 2016
      Mannequin

      New changeset d1b400943483 by Xavier de Gaye in branch '3.6':
      Issue bpo-20211: Do not add the directory for installing C header files and
      https://hg.python.org/cpython/rev/d1b400943483

      New changeset fcc9f19fcc13 by Xavier de Gaye in branch 'default':
      Issue bpo-20211: Merge 3.6.
      https://hg.python.org/cpython/rev/fcc9f19fcc13

    15. xdegaye commented on Dec 14, 2016

      @xdegaye
      xdegayemannequin
      on Dec 14, 2016
      Mannequin

      > this assumption is wrong for the multiarch case.

      Please explain why it is wrong.

      Sadly no explanations have been given, we have to take Matthias word for it.
      This new patch updates the add_multiarch_paths() method and includes LIBDIR and INCLUDEDIR in their corresponding search paths for the multiarch case.

    16. xdegaye
      removed their assignment
      on Dec 14, 2016

    17. xdegaye commented on Dec 14, 2016

      @xdegaye
      xdegayemannequin
      on Dec 14, 2016
      Mannequin

      Same patch, but this one is properly set to be handled by Rietveld.

    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    19. serhiy-storchaka commented on Dec 26, 2023

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 26, 2023
      Member

      setup.py no longer used to build CPython since 3.12 (see #94474).

    20. erlend-aasland
      added
      pendingThe issue will be closed if no feedback is provided
      on Dec 27, 2023
    21. hauntsaninja
      closed this as not plannedon Jan 7, 2024
    22. erlend-aasland
      removed
      pendingThe issue will be closed if no feedback is provided
      on Jan 7, 2024
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifebuildThe build process and cross-build

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions