Do not embed manifest files in *.pyd when compiling with MSVC

@malemburg

BPO	4120
Nosy	@malemburg, @loewis, @mhammond, @theller, @tiran, @skrah
Files	Python-2.6-no.manifest.in.pyd.diff: (committed) msvc9compiler_stripruntimes.diff: Strip the VC90 runtimes assembly from the dependencies in the manifest msvc9compiler_stripruntimes_revised.diff: revised patch msvc9compiler_stripruntimes_regexp.diff msvc9compiler_stripruntimes_regexp2.diff testpyd.py 3.1-byte-string.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2009-12-03.21:03:47.161>
created_at = <Date 2008-10-14.08:01:50.834>
labels = ['build', 'library']
title = 'Do not embed manifest files in *.pyd when compiling with\tMSVC'
updated_at = <Date 2010-04-26.19:31:14.754>
user = 'https://bugs.python.org/aheider'

bugs.python.org fields:

activity = <Date 2010-04-26.19:31:14.754>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = <Date 2009-12-03.21:03:47.161>
closer = 'loewis'
components = ['Build', 'Distutils']
creation = <Date 2008-10-14.08:01:50.834>
creator = 'aheider'
dependencies = []
files = ['11791', '11847', '15026', '15072', '15113', '15161', '15612']
hgrepos = []
issue_num = 4120
keywords = ['patch']
message_count = 52.0
messages = ['74723', '74730', '74742', '74747', '74758', '74763', '74835', '74926', '74930', '74992', '74997', '75035', '75071', '75072', '75081', '75153', '75155', '75159', '75183', '75194', '75567', '75570', '75579', '80148', '80908', '80909', '80910', '93440', '93441', '93458', '93700', '93702', '93713', '93716', '93906', '93913', '93943', '94140', '94202', '94221', '94644', '94660', '94669', '94673', '95853', '95946', '96641', '96769', '104237', '104238', '104244', '104268']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'mhammond', 'theller', 'christian.heimes', 'koen', 'aheider', 'zhirsch', 'jdetaeye', 'srid', 'cgohlke', 'skrah', 'ntouran', 'egaudry']
pr_nums = []
priority = 'high'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue4120'
versions = ['Python 3.1']

Instead of reverting the patch for bpo-2563, I propose to strip the
VC90 runtimes from the manifest (this will preserve other dependencies
needed). I checked this by editing tk85.dll and replacing the
dependency there with white-space - the Windows Common Controls
dependency still worked properly after this. I've attached a patch to
strip the runtime from the manifest when building Python extensions
(through search/replace).

I think somehow the manifests of the .pyd's in DLLs should also be
edited, because leaving them out will cause problems if this is done
when building tk85.dll. Or perhaps only special care is needed if there
ever is a .pyd with additional dependencies? All files with the .pyd
extension do not rely on anything else. I don't know enough about the
build process to suggest a change for this, except that the build
config for the .dll files inside DLLs might also need to be changed.

Skipping the mt.exe call seems fine to me, but there's another solution
which might be better:

The automatic manifest binding comes from a "#pragma
comment(linker,"/manifestdependency:" directive in crtdefs.h. That can
be disabled by setting _CRT_NOFORCE_MANIFEST. This way all other
manifest dependencies stay in tact and it could be set for *.pyd builds
(instead of my patch).

TCL's manifest is a different problem. I do not use tkinter, but
according to PCbuild/readme.txt it should be possible to get rid of the
CRT dependency the same way: just set _CRT_NOFORCE_MANIFEST in the nmake
line.

.. erm, the nmake line in Tools\buildbot\external.bat and
Tools/buildbot/external-amd64.bat ;)

_CRT_NOFORCE_MANIFEST sounds nice, but I'm having some trouble getting
it to work for Python extensions. I'm still having manifests
generated...? Can you show me how to use it? It is an undocumented
option.

I don't see a problem with this and can see how it would help with
private assemblies.

I tried to use _CRT_NOFORCE_MANIFEST but i couldn't get it working.
There're some infos about this approach at
http://blog.m-ri.de/index.php/2008/05/06/hotfix-fuer-usemsprivateassembliesh-und-vc-2008/
, but even with the mentioned "__declspec(selectany) int
_forceCRTManifest[RTM];" statement, i wasn't able to suppress the
manifest dependency :(

Microsoft, in its infinite wisdom, decided to compile the CRT itself
with those #pragma's enabled. That puts the "/manifestdependency" linker
arguments in msvcrt.lib. When you link against that file you get that
CRT manifest entry, whether you like it or not :(
Looks like _CRT_NOFORCE_MANIFEST only works if you build your own CRT
without the #pragma's.
Of course that only applies to the dynamic version of the CRT, when
using /MT the manifest hell just vanishes.

I'm unsure what the best way to fix this issue for python is...

OK, so the define is not going to work. For Python extensions built
through distutils, the msvc9compiler_stripruntimes.diff patch will
work: it will remove just the dependency on the VC90runtimes. It will
leave other assembly dependencies if they are there.

For the .pyd's and .dll's in the DLLs folder: I have opened them in a
hex editor, and just by hand replaced the <assemblyIdentity> entry for
the VC90runtimes in each file with spaces. I can confirm that this
works, also for _tkinter, for which tk85.dll has an additional
dependency on Windows Common Controls (though this one was available in
my WinSXS). What would be the best way to integrate a search/replace
operation like this in the Python build process? Is the manifest
generated as a seperate file here first as well, or is it directly
embedded?

What would be the best way to integrate a search/replace
operation like this in the Python build process? Is the manifest
generated as a seperate file here first as well, or is it directly
embedded?

What's wrong with Andre Heider's patch?

Ah, I wasn't thinking it through. It is fine for all .pyd of course,
because they don't have any extra dependencies. I was thinking of DLLs
with extra dependencies (of which there is one, TK85.dll). There,
leaving out the manifest will break it. But indeed, the patch is fine
for .pyd building. My mistake!

Should this patch be applied to 3.0 before the next RC lands? Barry
hasn't released RC2 yet.

I have now committed Python-2.6-no.manifest.in.pyd.diff as r67120,
r67121, and r67122. Thanks for the patch.

In r67125, r67126, r67127, r67128, r67129, I removed the mt.exe
invocation from tcl and tk, and removed the DLLs CRT manifest from the
installer.

Without msvc9compiler_stripruntimes.diff, or a patch like it, it doesn't
seem possible to use pywin32 (or other binary extensions) with
Python-2.6+ installed "just for me" (i.e., without the VC90 runtime
installed globally).

Using the python-2.6.1 MSI installed "just for me" and the pywin32-212
installer, I get the following error when trying to import win32api:

    Python 2.6.1 (r261:67517, Dec  4 2008, 16:51:00) [MSC v.1500 32 bit
(Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import win32api
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: DLL load failed: This application has failed to start
because the application configuration is incorrect. Reinstalling the
application may fix this problem.

Removing the manifest from all the .pyd files installed by pywin32
allows me to import win32api without issue.

Is the report posted in:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/3ec
6af1279a162ca#
also a symptom of the same problem?

Johan

@johan

@johan: no it is not a symptom of the same problem. The problem you are
having is that the executable which is embedding Python does not have a
manifest. If you add a manifest with a dependency on the VC90 runtimes
to the main executable, then it should work AFAIK.

@Koen

@Koen: You're aboslutely right. The issue is solved indeed by adding a
dependency in the manifest.

Could msvc9compiler_stripruntimes.diff still be considered for
inclusion in Python 2.6.3 (high priority?)?
A lot of .pyd's of third-party extensions are getting dependencies on
MSVCR90 added to the .pyd, which makes them not work if the runtimes
are not installed into the WinSxS folder and/or when embedding into
other applications. For example, PIL encountered this problem recently.
In essence, all Windows Python extensions built using setup.py
build_ext encounter this problem.
What the patch does is edit the manifest file to remove only the
dependency on Microsoft.VC90.CRT.

It probably won't make 2.6.3 as the final is just out. But can we have
it for 2.6.4?
Here's a link to the relevant discussion on ImageSIG: http://
mail.python.org/pipermail/image-sig/2009-October/005918.html

Looking at my local extensions installed, Matplotlib has applied this
patch, but PyGame and PIL have embedded manifests with dependencies on
the runtime in their .pyd's.

The patch now does a text search/replace on an XML file, if you think
this is bad style I could rewrit it to use proper XML parsing and
writing.

There are two easy to fix issues with the
msvc9compiler_stripruntimes.diff patch: 1) the dependency for 64-bit
VC90.CRT is not removed and 2) the VC90.CRT dependency is removed also
from executables, which will fail to run. A revised patch is attached.
This is used to build the Matplotlib binaries for Windows and also works
for numpy, pygame and PIL.

Thanks Christoph, those are two important fixes to the patch. I'm +1 on
having this in the next 2.6 maintenance release.

Just a note on the style of the msvc9compiler_stripruntimes_revised.diff
patch:

please try to use lines with at most 80 chars only
it would be better to use regexps to do the search&replace and
perhaps add wildcards to catch future SP-versions of VC9 as well

Added Distuils as component, since the patch is targeting distutils and
extensions, not the Python build process.

The attached patch uses a regular expression.

Christoph Gohlke wrote:

Christoph Gohlke <cgohlke@uci.edu> added the comment:

The attached patch uses a regular expression.

Much better, thanks.

Looking at the description of manifest files, it appears that just
removing the assemblyIdentity-element results in an invalid manifest file:

http://msdn.microsoft.com/en-us/library/aa374219(VS.85).aspx

It appears that the entire dependency-element referencing the MS VC90
CRT DLL has to be removed in order to make the manifest correct again.

Looking at the schema, it's enough to just remove the
dependentAssembly-element:

http://msdn.microsoft.com/en-us/library/aa375635(VS.85).aspx

Reading up on the manifest trouble-shooting page at (near the end of the
page):

http://msdn.microsoft.com/en-us/library/ms235342.aspx

it may actually be enough to just place the Microsoft.VC90.CRT.manifest
file into the Python folder (the one with python.exe and the CRT DLLs):

"""
If the operating system fails to find the CRT or any other assembly as a
shared assembly, it starts looking for the assembly as a private
assembly. It searches for private assemblies in the following order:

Check the application local folder for a manifest file with name
<assemblyName>.manifest. In this example, the loader tries to find
Microsoft.VC90.CRT.manifest in the same folder as appl.exe. If the
manifest is found, the loader loads the CRT DLL from the application
folder. If the CRT DLL is not found, load fails.
"""

This patch also removes empty dependentAssembly elements after removing
the VC.CRT assemblyIdentity element.

It seems not enough to just place the Microsoft.VC90.CRT.manifest and VC
runtime DLL files into the Python folder. On a system without the VC
runtime installed in winsxs, trying to import a pyd module build without
the patch fails with "ImportError: DLL load failed: ...". Import works
when 1) the manifest and VC runtime DLLs are also placed in the same
folder as the pyd file, or 2) the Microsoft Visual C++ 2008
redistributable package is installed, or 3) the pyd file is build with
the patch installed. Tested with Windows XP SP3, Python 2.6.3,
matplotlib 0.99.1, Visual Studio 2008 SP1.

It may actually be enough to just place the
Microsoft.VC90.CRT.manifest file into the Python folder (the one with
python.exe and the CRT DLLs).

I concur with what Christoph says, that is how the embedded
installation works. However, the .pyd files are not in the Python main
folder. And therefore they shouldn't have a manifest for the runtimes
(because it will look in the current folder relative to the DLL/pyd),
because then the runtimes of the main python.exe will be used (which
can then be in the same folder as the 'embedded' MSVCR location).

Just to share my recent experience with this issue: I was attempting to
get Python 2.6 working with py4mpi, matplotlib, and pymssql on a 64-bit
Windows Vista based HPC cluster via x-copy deployment without the charm
of administrative privileges on the server and on my workstation (which
is XP). It wasn't easy, but I got it working.

I attempted to apply the patch listed here and recompile the packages
that complained about Side-by-side configurations, but got in too deep
while compiling matplotlib on windows (recall I don't have admin
privileges and can't install the prereqs) so I dug deeper.

I found this: http://blog.kalmbachnet.de/?postid=80 which suggests
removing the publicKeyToken attribute from the manifests to force local
DLLs. This would possibly give the same effect as not embedding the
manifests in the first place using the patch. So I went in, using
VS2008, and removed this attribute from the embedded manifests in
python26.dll, python.exe, and the manifest file in C:\python26. I also
copied msvcm90.dll, msvcp90.dll, and msvcr90.dll from the 9.0.21022.8
folder in c:\Windows\WinSxS into c:\Python26. With that, matplotlib
started working, but pymssql still did not.

I then renamed _mssql.pyd to _mssql.dll so that VS2008 could recognize
the manifest, removed the publicKeyToken attribute, and renamed it back
to _mssql.pyd, but it still complained. Finally, using depends.exe from
the internet, I noticed msvcr71.dll was required on my local machine, so
I tried copying it over to the remote machine, into the site-packages
folder. Bam. It worked. So while I don't really consider this a
solution, it definitely worked for what I needed. Any other 3rd party
modules that complain in the future in my weird xcopy-deployment will
undergo manifest-stripping via VS2008 and other dependency checking via
depends.exe. Yay.

By the way, the pymssql was compiled with VS2005, which is why it
requires msvcr71.dll. If I could rebuild it with VS2008, I imagine that
step wouldn't be necessary. While I do have VS2008 on hand, I don't have
MS SQL 2020 Super Awesome Developer Edition, or whatever 3GB package
installs the required header.

What's a procedure for testing this patch (please be as precise as
possible)?

There are two levels of testing.

First, on a Windows development system with Visual Studio 2008, Python
2.6.2+, and the msvc9compiler_stripruntimes_regexp2.diff patch applied,
verify that the embedded manifest in distutil generated PYD extension
files is stripped of the WinSxS VC.CRT dependency and that the
extensions can still be imported. The attached script (testpyd.py)
builds a minimal extension (testpyd.pyd), searches for the WinSxS VC.CRT
dependency within the pyd file, and imports the extension. The script
should run without exception.

Second, verify that distutil generated PYD extensions can also be
imported on a test system, which has no Visual Studio 2008 runtime
installed in %WINDIR%\WinSxS:
Install Windows XP SP3, preferably in a virtual machine. Do not install
any additional programs or patches, which might install the Visual
Studio 2008 runtime. Then install Python 2.6.2+ using the "Install just
for me" option. Try import the testpyd extension built on the
development system: run "python.exe -c 'import testpyd'" in a directory
containing the testpyd.pyd file (not the Python installation directory
containing python.exe). This should work without exception.

Importing the testpyd extension built without the patch fails on the
test system:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: This application has failed to start
because the application configuration is incorrect. Reinstalling the
application may fix this problem.

Tested with Python 2.6.3 for Windows 32-bit.

Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with "ImportError: DLL load failed:..." if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).

/* Import the testpyd.pyd module. */
#include <Python.h>
int main(void) {
    Py_Initialize();
    { 
        PyObject *p = PyImport_ImportModule("testpyd"); 
    }
    Py_Finalize();
    return 0;
}

Christoph Gohlke wrote:

Christoph Gohlke <cgohlke@uci.edu> added the comment:

Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with "ImportError: DLL load failed:..." if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).

/* Import the testpyd.pyd module. */
#include <Python.h>
int main(void) {
Py_Initialize();
{
PyObject *p = PyImport_ImportModule("testpyd");
}
Py_Finalize();
return 0;
}

I'm not sure whether this is related to the problem in question.
Does MinGW also embed a manifest in the generated executable ?

If it doesn't, then this is more likely a problem with how
MinGW works per default, than with Python or distutils.

The fact that the above does work without the patch applied
suggests that MinGW does not embed the manifest in the executable
(since the Windows linker then uses the one from the PYD file).

My last comment was merely reporting that this patch can break MinGW
based build processes. It did surprise some developers that their
programs embedding matplotlib, which is now build with the MSVC9 patch,
stopped working.

The MinGW breakage probably comes from the same "issue" as encountered
in http://bugs.python.org/issue4120#msg80908 and #msg80909: the main
application, which embeds Matplotlib, does not have a manifest for the
MSVCR9 runtimes? That's a problem which should be addressed by adding a
manifest in the main application (and manifests can also be separate
files called app.exe.manifest if you do not want to embed it).

I came across this (very) interesting thread after experiencing some
redistribution issue with a python(2.6)-based package built with msvc9
compiler.

I used to struggled with the SxS Microsoft policy in the past. After I
finally went for the private assembly/crt runtime redistribution
solution, I always succeeded to deliver a stand-alone package to any
windows host around (I got full control on what's actually build and
distribute).

Having now (using the disutils module) the manifest file embedded in the
python extension actually forbids this redistribution solution, for the
reasons that have been posted here (SxS policy), unless a private
assembly/crt runtime is provided next to each built extension. IMHO,
this is not a convenient (and common) way to redistribute a software.

This is why I fully agree with the propositions that were made here,
i.e. not embedding manifest into a python extension built with the
distutils module and msvc.

Could anyone tell us if a decision has been made about such a change ?

Thanks,
Eloi

Thanks for the patch. Committed as r76651, r76652, r76653, r76654,
r76655 and r76656.

With the latest Python3.1 svn version of msvc9compiler.py, I get a
"can't use a string pattern on a bytes-like object" error. The
attached diff fixes the error.

See bpo-7556 for the "can't use a string pattern on a bytes-like object"
error.

I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

First question: Open a new issue, or discuss it in this one (and repoen it)?

Hi Thomas,

I think we should open a new issue (child from bpo-4120), just to make
sure the whole thing remains understandable.

Regards,
Eloi

Thomas Heller wrote:

Thomas Heller <theller@ctypes.org> added the comment:

I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

First question: Open a new issue, or discuss it in this one (and repoen it)?

----------
nosy: +theller

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4120\>

--

Eloi Gaudry

Free Field Technologies
Axis Park Louvain-la-Neuve
Rue Emile Francqui, 1
B-1435 Mont-Saint Guibert
BELGIUM

Company Phone: +32 10 487 959
Company Fax: +32 10 454 626

I mentioned the problem with pythoncom.dll and suggested a solution in bpo-7833.

First question: Open a new issue, or discuss it in this one (and repoen it)?

Please open a new issue. AFAICT, the original issues request (do not
embed manifests in pyd files) is now implemented fully. Feel free to
leave a link to the new issue here, but please start over describing the
problem you are seeing, and in what version you are seeing it.

Navigation Menu

Repository navigation

Do not embed manifest files in *.pyd when compiling with MSVC #48370

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Do not embed manifest files in *.pyd when compiling with MSVC #48370

Description

Activity

koen commented on Oct 21, 2008

aheider commented on Oct 22, 2008

aheider commented on Oct 22, 2008

koen commented on Oct 22, 2008

mhammond commented on Oct 23, 2008

aheider commented on Oct 24, 2008

koen commented on Oct 24, 2008

loewis commented on Oct 24, 2008

koen commented on Oct 24, 2008

tiran commented on Nov 6, 2008

loewis commented on Nov 6, 2008

loewis commented on Nov 6, 2008

zhirsch commented on Jan 19, 2009

jdetaeye commented on Feb 1, 2009

koen commented on Feb 1, 2009

jdetaeye commented on Feb 1, 2009

koen commented on Oct 2, 2009

koen commented on Oct 2, 2009

cgohlke commented on Oct 2, 2009

koen commented on Oct 7, 2009

malemburg commented on Oct 7, 2009

cgohlke commented on Oct 7, 2009

malemburg commented on Oct 7, 2009

malemburg commented on Oct 12, 2009

cgohlke commented on Oct 13, 2009

koen commented on Oct 13, 2009

ntouran commented on Oct 16, 2009

loewis commented on Oct 18, 2009

cgohlke commented on Oct 18, 2009

cgohlke commented on Oct 29, 2009

malemburg commented on Oct 29, 2009

cgohlke commented on Oct 29, 2009

koen commented on Oct 29, 2009

egaudry commented on Dec 1, 2009

loewis commented on Dec 3, 2009

skrah commented on Dec 19, 2009

srid commented on Dec 21, 2009

theller commented on Apr 26, 2010

egaudry commented on Apr 26, 2010

cgohlke commented on Apr 26, 2010

loewis commented on Apr 26, 2010

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions