Repository navigation
64bit cross compilation on windows #46765
Description
| BPO | 2513 |
|---|---|
| Nosy | @malemburg, @loewis, @mhammond, @theller, @tiran, @tpn |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/mhammond'
closed_at = <Date 2008-04-07.02:00:19.240>
created_at = <Date 2008-03-30.07:01:23.890>
labels = ['type-feature', 'library']
title = '64bit cross compilation on windows'
updated_at = <Date 2008-04-09.18:57:01.455>
user = 'https://github.com/mhammond'bugs.python.org fields:
activity = <Date 2008-04-09.18:57:01.455>
actor = 'loewis'
assignee = 'mhammond'
closed = True
closed_date = <Date 2008-04-07.02:00:19.240>
closer = 'mhammond'
components = ['Distutils']
creation = <Date 2008-03-30.07:01:23.890>
creator = 'mhammond'
dependencies = []
files = ['9900', '9914']
hgrepos = []
issue_num = 2513
keywords = ['patch', '64bit']
message_count = 14.0
messages = ['64743', '64790', '64825', '64826', '64828', '64834', '64835', '64836', '64843', '64849', '64860', '64884', '65059', '65257']
nosy_count = 6.0
nosy_names = ['lemburg', 'loewis', 'mhammond', 'theller', 'christian.heimes', 'trent']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2513'
versions = ['Python 2.6']
Activity
mhammond commentedon Mar 30, 2008
I've taken the liberty of adding Trent, Christian and Martin to the nosy
list as I know they are actively, if reluctantly interested in this.
This patch allows the distutils to cross-compile on Windows. It has
been tested on x86 and amd64 platforms, with both platforms successfully
able to natively and cross-compile extension modules and create binary
distributions.
To cross-compile, specify '--plat-name' to the build command (valid
values are 'win32', 'win-amd64' and 'win-ia64'). This option name was
chosen to be consistent with the bdist_dumb command. I've included the
docs I added below (which are also in the patch), but note that as with
native compilation using distutils, it's not necessary to set any
environment variables or do anything else special with your environment
to make this work.
The patch also adds a x64 target for the 'bdist_wininst' target, which
it creates as distutils/command/wininst-9.0-amd64.exe. This executable
is necessary even for bdist_wininst to work natively on x64, but is
still included here for simplicity.
To assist with testing, I've also added a distutils setup.py script to
the PC/example_nt directory. This is capable of creating bdist_wininst
executables for both native and cross platforms; 'setup.py build
--platname=win-amd64 bdist_wininst' will create an amd64 installer on an
x86 machine.
The patch has not been tested with a Visual Studio environment without
cross-compile tools installed - it will obviously fail, but its not
clear how ugly this failure will be.
Below is the text I added to docs/distutils/builtdist.rst:
Cross-compiling on Windows
=====================
Starting with Python 2.6, distutils is capable of cross-compiling
between Windows platforms. In practice, this means that with the
correct tools installed, you can use a 32bit version of Windows to
create 64bit extensions and vice-versa.
To build for an alternate platform, specify the :option:`--plat-name`
option to the build command. Valid values are currently 'win32',
'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows,
you could execute::
to build a 64bit version of your extension. The Windows Installers
also support this option, so the command::
would create a 64bit installation executable on your 32bit version of
Windows.
Note that by default, Visual Studio 2008 does not install 64bit
compilers or tools. You may need to reexecute the Visual Studio setup
process and select these tools.
theller commentedon Mar 31, 2008
I had to make additional changes to PCBuild\pcbuild.sln to create a
64-bit wininst-9.0-amd64.exe (but I was not able to try out if it works
or not).
malemburg commentedon Apr 1, 2008
Hi Mark,
I hope this patch makes it into 2.6.
One thing I'm curious about is the version of "Visual Studio 2008" that
is needed to support this kind cross-compilation and which version will
be or is used to build the Python versions on 64-bit Windows.
malemburg commentedon Apr 1, 2008
Nevermind, diving deep into the MSDN, I found the answer:
http://msdn2.microsoft.com/en-us/library/hs24szh9.aspx
So for cross-compilation, the Visual Studio 2008 Standard Edition is
sufficient.
Is the Professional Edition used for the standard Python binaries ? Does
it generate faster code ?
A bit off-topic, I know, but it's really hard to find this important
information elsewhere and may well be of use for other Python extension
authors as well.
Thanks.
mhammond commentedon Apr 1, 2008
Hi Marc-Andre,
The PCBuild/README.txt file has some info about compatibility with VS
versions (but probably needs to say more about x64 builds). There is
also talk about releasing "Profile Guided Optimization" built binaries,
which does require a professional version. I believe, but am not sure,
that Martin uses the professional versions to make the official
binaries. Either way, this might be getting off-topic for this patch.
tpn commentedon Apr 2, 2008
Hi guys, have been on holiday since PyCon ended, only getting back into
the swing of things now. With regards to the x64 Windows build,
indeed, PCbuild/readme.txt is definitely in need of an update,
especially with the details of the slightly new build system I checked
in a few weeks ago.
As it currently stands, the Professional version of Visual Studio 2008
is required for a complete x64 build, however, this is because a cross-
compilation 'vsvars32.bat'-type file needs to be called in order to do
command line (i.e. nmake) x64 builds.
(The particular file I'm referring to is "C:\Program Files (x86)
\Microsoft Visual Studio 9.0\VC\vcvarsall.bat", which needs to be
invoked with 'x86_amd64' as the first parameter. This is only shipped
with the Professional version of Visual Studio 2008 -- the buildbots
currently rely on it to produce x64 builds, see
http://svn.python.org/projects/python/trunk/Tools/buildbot/build-
amd64.bat.)
We need to invoke this particular batch file in order to build x64
versions tcl/tk and openssl because we're currently reliant on the
nmake-oriented builds each of these projects ship with.
That said, the next step in my build-related improvements is to remove
this reliance and mimick the nmake-oriented builds with custom .vcproj
builds that inherit all of our property sheets -- this is the approach
I took with bsddb and sqlite3 and it worked *very*, *very* well. (See
_bsddb44.vcproj and sqlite3.vcproj in the PCbuild directory.)
There are many advantages with this approach. To name a few: the
release versions of the modules will benefit from profile guided
optimisation, just like the rest of the python build, the modules are
significantly easier to debug, and there is no chance for conflicting
compiler/linker flags (which was a continual source of pain with
bsddb/sqlite3 when they were built with the vendor-supplied build
system).
So, eventually, you should be able to build and entire release version
of Python x64 with just the Express edition of Visual Studio 2008 (note
that the Express edition won't support a PGO build though).
The 'official' release though will be built with a Professional version
though, as Mark states, thus allowing us to take advantage of profile
guided optimisation.
Hopefully this isn't too much off topic. (I can't comment on the
patches Mark has provided just yet as I haven't had a chance to review
them -- just wanted to clear up the x64 cross-compilation questions.)
Perhaps I should create a separate issue to track the work needed to
cross compile an x64 build with the Express edition only. Any
objections?
ctheune commentedon Apr 2, 2008
Mark, I think you mixed me up with Christian Heimes, so I'm removing
myself here (at least for now).
tpn commentedon Apr 2, 2008
Humm, just saw the MSDN page Marc-Andre referred to, which indeed seems
to indicate the Express edition doesn't support x64 cross-compilation
*at all*. AFAICT the latest Platform SDK (6.1) ships with an x64 cross
compilation environment, but it's command line only, so unless there's
a way to hook it up such that 'vcbuild' uses it with the Express
edition as well -- it seems like you do need Standard edition or higher
for x64 builds.
tiran commentedon Apr 2, 2008
Could somebody with a 64bit CPU please extend the PCbuild/README.txt? I
don't have the means to test 64bit builds or cross compilation of a
32bit build on a 64bit system. I only have 32bit CPUs at home.
+1 for nmake if you can get it to work.
A while ago I suggested to ship both 32bit and 64bit lib files with
every flavor of Python. With both library sets developers could cross
compile Windows extensions. Comments?
tpn commentedon Apr 2, 2008
Christian, regarding shipping x64 .lib files as well as 32-bit ones --
sensible idea, but where would we place the x64 version libs? In Python
[xx]\libs\amd64? (This would mirror the approach used with PCbuild and
PCbuild\amd64 -- although there's currently an ongoing discussion on
this topic in python-dev.)
With regards to updating PCbuild/readme.txt, I'm planning on doing that
by EOD today.
tpn commentedon Apr 2, 2008
Updated PCbuild/readme.txt in r62105 and r62106 for trunk and py3k
respectively.
mhammond commentedon Apr 2, 2008
I'd like to keep this issue specifically about cross-compilation in the
current tree, and while some of the other ideas may have merit, let's
not bog this issue down with them unless they directly impact the patch.
Does anyone have any comments or concerns about my patch? Anything
people would like to see changed before I check it in?
Martin - my patch creates a new executable in Lib\distutils\command, and
IMO it should be included in binary builds for all windows platforms.
Is there something I should change in the MSI process?
mhammond commentedon Apr 7, 2008
Checked in r62197, including the patch from Thomas. Note I expanded the
patch to the distutils doc (noting you need to build Python itself for
the target platform before it works) and added a short entry to Misc/NEWS
loewis commentedon Apr 9, 2008
Yes, I fixed that in r62255: it now packages wininst*.exe from that
directory.
Regards,
Martin