Do not embed manifest files in *.pyd when compiling with MSVC

@malemburg

BPO	4120
Nosy	@malemburg, @loewis, @mhammond, @theller, @tiran, @skrah
Files	Python-2.6-no.manifest.in.pyd.diff: (committed) msvc9compiler_stripruntimes.diff: Strip the VC90 runtimes assembly from the dependencies in the manifest msvc9compiler_stripruntimes_revised.diff: revised patch msvc9compiler_stripruntimes_regexp.diff msvc9compiler_stripruntimes_regexp2.diff testpyd.py 3.1-byte-string.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2009-12-03.21:03:47.161>
created_at = <Date 2008-10-14.08:01:50.834>
labels = ['build', 'library']
title = 'Do not embed manifest files in *.pyd when compiling with\tMSVC'
updated_at = <Date 2010-04-26.19:31:14.754>
user = 'https://bugs.python.org/aheider'

bugs.python.org fields:

activity = <Date 2010-04-26.19:31:14.754>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = <Date 2009-12-03.21:03:47.161>
closer = 'loewis'
components = ['Build', 'Distutils']
creation = <Date 2008-10-14.08:01:50.834>
creator = 'aheider'
dependencies = []
files = ['11791', '11847', '15026', '15072', '15113', '15161', '15612']
hgrepos = []
issue_num = 4120
keywords = ['patch']
message_count = 52.0
messages = ['74723', '74730', '74742', '74747', '74758', '74763', '74835', '74926', '74930', '74992', '74997', '75035', '75071', '75072', '75081', '75153', '75155', '75159', '75183', '75194', '75567', '75570', '75579', '80148', '80908', '80909', '80910', '93440', '93441', '93458', '93700', '93702', '93713', '93716', '93906', '93913', '93943', '94140', '94202', '94221', '94644', '94660', '94669', '94673', '95853', '95946', '96641', '96769', '104237', '104238', '104244', '104268']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'mhammond', 'theller', 'christian.heimes', 'koen', 'aheider', 'zhirsch', 'jdetaeye', 'srid', 'cgohlke', 'skrah', 'ntouran', 'egaudry']
pr_nums = []
priority = 'high'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue4120'
versions = ['Python 3.1']

The MSVC build process currently embeds the .manifest file, which is
generated by the MS linker, in the following files:

python.exe
pythonXY.dll
*.pyd

The latter is problematic on machines without the MS CRT redistributable
installed (No CRT files in %WINDIR%\WinSxS). While this error won't
occur when the python package is installed, it will in other cases like
in ours:

We use the python interpreter within our application and do ship
python25.dll. We also ship the MS CRT files and place them next to the
main application (MS calls this "private assemblies" and "xcopy
deployment). We do not install the redistributable, because it needs
admin rights.
If a user tries to "import socket" on this setup it will fail, because
the Side by Side / Fusion loader expects the CRT next to the calling
Binary (%PYTHONHOME%\DLLs\_socket.pyd) because of the manifest.

The solution is to not embed the manifest in the *.pyd Modules. This way
the modules just link against msvc*.dll. This works because the windows
loader has already mapped the CRT into memory for the process (either
through python.exe or pythonXY.dll). This is also the only reliable way
i could find to fix the import on machines without redist.

Attached you'll find a patch for the MSVC90 build, apply with -p5. It
will only patch the release file, PCbuild8/pyd_d.vsprops requires the
same fix.

Trolltech also uses this approach for its plugins:
https://trolltech.com/developer/faqs/faq.2007-10-19.6185511205

I'm skeptical about this patch. It seems like it could produce an
incompatibility with existing installations or deployment procedures. In
any case, the VS 2008 project files are not officially supported.

If VS 2005 is any similar with VS 2008 wrt. to manifests, I think you
can solve your problem by providing a CRT manifest in the directory that
has _socket.pyd.

I'd be rather interested in seeing the consequences of this approach for
Python 2.6.

Yes, i know that v2.5 doesn't officially support MSVC9. But the same
problem applies to MSVC8 and its CRT. But in contrast to MSVC9, the CRT8
redist is installed on almost every machine, because alot of software
installs it. But thats basically just luck ;)

Placing just a .manifest next to *.pyd won't work either. If you're
going this route, you have to put the 3 CRT DLLs there too (and thats
actually the approach mentioned here:
http://msdn.microsoft.com/en-us/library/ms235291.aspx)
Then you've got 2 cases:

The user does have the redist installed
or
He does not

for 1) The WinSXS installation has a higher priority and the local files
are ignored
for 2) The files next to the binaries are used (eg c:\python and
c:\python\DLLs)

Sounds fine, but the latter raises another issue: Now you have 2 copies
of the DLLs. Based on these 2 unique file names, windows decides to load
2 copies into memory, one for python.exe|dll and one for *.pyd. As you
might see, this gets very nasty because you have 2 different heaps now.
Allocating memory from the one instance and freeing it in the other
makes the whole process crash.

This issue doesn't stop here, here's another situation:
Python is compiled with MSVC9. A user has it and MSVC9 SP1 installed,
which has a newer CRT version. If he builds site-packages, these files
have an embedded manifest with another CRT version than the python
interpreter itself. Importing these packages loads a second copy of a
CRT into the processes memory, just like described above.

I don't know if or for what version this should be addressed for python.
I'm just saying that this was a huge issue the last days at the company
i work for.

Yes, i know that v2.5 doesn't officially support MSVC9. But the same
problem applies to MSVC8 and its CRT.

The point is that even the MSVC8 project files are not supported.
They have been included, but they are not actually used for anything.

Yes, i know that v2.5 doesn't officially support MSVC9. But the same
problem applies to MSVC8 and its CRT.

Not necessarily. Take a look at how I deploy Python 2.6. I use a single
copy of the DLL, but two copies of the manifest (both referring to the
same DLL image). AFAICT, this works fine on XP (but fails on Vista SP1,
which complains that the manifest is ill-formed).

I'm closing this for 2.5 as rejected; it might cause more problems than
it solves, and 2.5.3 will be the last release (i.e. with no release to
fix it if it breaks something badly).

If you can come up with a working patch for 2.6: that would be more
interesting. Would the many manifest files that get generated need to be
shipped as well?

Ok, point taken, so lets aim at v2.6. Should i open a new issue then?

Attached you'll find a new diff against v2.6. This time pyd_d.vsprops
gets patched too and the linker doesn't even generate a .manifest file.
So no, the *.pyd files do not get any manifest info, neither a loose
*.pyd.manifest nor an embedded one.

I checked a per-user v2.6 install with the CRT manifest pointing at
"..". I also tried this approach for our software, but i couldn't get it
working on w2k3.

The solution i am proposing doesn't need a CRT manifest file in
"./DLLs". This approach works on w2k3, but i can not test this on vista SP1.

I'm just repurposing the issue (despite what I usually claim as a policy)

To my surprise there indeed is a vista SP1 box in our test farm. We
tested my patch on the following windows machines (each without an
installed CRT redist):

xp SP2 and SP3
xp64 SP2
server 2003 R2 SP1
vista with and without SP1
windows server 2008

For some reason, the linker *does* generate manifest files for all the
.pyds when I try it out. Should that worry me? When I delete them, it
still can import them just fine.

FWIW, with the patch alone, the manifest in .DLLs will still be needed,
to be able to load tcl85.dll.

In response to 74742:

This issue doesn't stop here, here's another situation:
Python is compiled with MSVC9. A user has it and MSVC9 SP1 installed,
which has a newer CRT version. If he builds site-packages, these files
have an embedded manifest with another CRT version than the python
interpreter itself. Importing these packages loads a second copy of a
CRT into the processes memory, just like described above.

This is not true. For MSVC9SP (VS2008) Microsoft decided that by
default, it will still compile against the original CRT. Only by
explicitly adding a compiler flag can you use the updated SP1 CRT (they
did this because the 2005SP1 caused so much grief, which did
automatically upgrade the CRT, and subsequently companies did not roll
out SP1 to prevent the mixing of CRTs). So you can safely compile using
SP1 installed, and you will at least still use the same CRT version.

For some reason, the linker *does* generate manifest files for all the
.pyds when I try it out. Should that worry me? When I delete them, it
still can import them just fine.

Hm, works for me. I am using MS SDK v6.1 and PCbuild/build.bat.

This is not true. For MSVC9SP (VS2008) Microsoft decided that by
default, it will still compile against the original CRT

You're right, there are defines for that:
http://msdn.microsoft.com/en-us/library/cc664727.aspx

While that won't load a 2nd copy of the CRT into memory, the embedded
manifests of site-packages still have the same problem as the .pyd files
now. So if this gets accepted, reverting the patch of bpo-2563 should
be considered.

The TCL85.dll introduces a subtlety (actually it is TK85.dll). The
MSVCRT90 is not the only dependency that can be in the manifest. The
Windows.CommonControls.6.0 is used by TK85.dll. So, also for all Python
extensions which have additional dependencies beside MSVCRT90, the
manifest should still be included.

Situation:

.pyd only depends on MSVCRT90: no manifest needed (will fallback to
Python 2.6's own manifest and to WinSXS or the installation in the main
Python folder)
.pyd has more dependencies: manifest is needed. But, a wild idea,
perhaps we could strip MSVCRT90 from the manifest? This is speculation,
but perhaps the fallback to the manifest of Python26.exe will still
work this way, and the runtimes will only need to be in a single place.

Anyone know of an easy-to-build example of a Python extension with
extra dependencies?

There are two levels of testing.

First, on a Windows development system with Visual Studio 2008, Python
2.6.2+, and the msvc9compiler_stripruntimes_regexp2.diff patch applied,
verify that the embedded manifest in distutil generated PYD extension
files is stripped of the WinSxS VC.CRT dependency and that the
extensions can still be imported. The attached script (testpyd.py)
builds a minimal extension (testpyd.pyd), searches for the WinSxS VC.CRT
dependency within the pyd file, and imports the extension. The script
should run without exception.

Second, verify that distutil generated PYD extensions can also be
imported on a test system, which has no Visual Studio 2008 runtime
installed in %WINDIR%\WinSxS:
Install Windows XP SP3, preferably in a virtual machine. Do not install
any additional programs or patches, which might install the Visual
Studio 2008 runtime. Then install Python 2.6.2+ using the "Install just
for me" option. Try import the testpyd extension built on the
development system: run "python.exe -c 'import testpyd'" in a directory
containing the testpyd.pyd file (not the Python installation directory
containing python.exe). This should work without exception.

Importing the testpyd extension built without the patch fails on the
test system:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: This application has failed to start
because the application configuration is incorrect. Reinstalling the
application may fix this problem.

Tested with Python 2.6.3 for Windows 32-bit.

Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with "ImportError: DLL load failed:..." if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).

/* Import the testpyd.pyd module. */
#include <Python.h>
int main(void) {
    Py_Initialize();
    { 
        PyObject *p = PyImport_ImportModule("testpyd"); 
    }
    Py_Finalize();
    return 0;
}

Christoph Gohlke wrote:

Christoph Gohlke <cgohlke@uci.edu> added the comment:

Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with "ImportError: DLL load failed:..." if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).

/* Import the testpyd.pyd module. */
#include <Python.h>
int main(void) {
Py_Initialize();
{
PyObject *p = PyImport_ImportModule("testpyd");
}
Py_Finalize();
return 0;
}

I'm not sure whether this is related to the problem in question.
Does MinGW also embed a manifest in the generated executable ?

If it doesn't, then this is more likely a problem with how
MinGW works per default, than with Python or distutils.

The fact that the above does work without the patch applied
suggests that MinGW does not embed the manifest in the executable
(since the Windows linker then uses the one from the PYD file).

My last comment was merely reporting that this patch can break MinGW
based build processes. It did surprise some developers that their
programs embedding matplotlib, which is now build with the MSVC9 patch,
stopped working.

The MinGW breakage probably comes from the same "issue" as encountered
in http://bugs.python.org/issue4120#msg80908 and #msg80909: the main
application, which embeds Matplotlib, does not have a manifest for the
MSVCR9 runtimes? That's a problem which should be addressed by adding a
manifest in the main application (and manifests can also be separate
files called app.exe.manifest if you do not want to embed it).

I came across this (very) interesting thread after experiencing some
redistribution issue with a python(2.6)-based package built with msvc9
compiler.

I used to struggled with the SxS Microsoft policy in the past. After I
finally went for the private assembly/crt runtime redistribution
solution, I always succeeded to deliver a stand-alone package to any
windows host around (I got full control on what's actually build and
distribute).

Having now (using the disutils module) the manifest file embedded in the
python extension actually forbids this redistribution solution, for the
reasons that have been posted here (SxS policy), unless a private
assembly/crt runtime is provided next to each built extension. IMHO,
this is not a convenient (and common) way to redistribute a software.

This is why I fully agree with the propositions that were made here,
i.e. not embedding manifest into a python extension built with the
distutils module and msvc.

Could anyone tell us if a decision has been made about such a change ?

Thanks,
Eloi

Thanks for the patch. Committed as r76651, r76652, r76653, r76654,
r76655 and r76656.

With the latest Python3.1 svn version of msvc9compiler.py, I get a
"can't use a string pattern on a bytes-like object" error. The
attached diff fixes the error.

See bpo-7556 for the "can't use a string pattern on a bytes-like object"
error.

I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

First question: Open a new issue, or discuss it in this one (and repoen it)?

Hi Thomas,

I think we should open a new issue (child from bpo-4120), just to make
sure the whole thing remains understandable.

Regards,
Eloi

Thomas Heller wrote:

Thomas Heller <theller@ctypes.org> added the comment:

I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

First question: Open a new issue, or discuss it in this one (and repoen it)?

----------
nosy: +theller

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4120\>

--

Eloi Gaudry

Free Field Technologies
Axis Park Louvain-la-Neuve
Rue Emile Francqui, 1
B-1435 Mont-Saint Guibert
BELGIUM

Company Phone: +32 10 487 959
Company Fax: +32 10 454 626

I mentioned the problem with pythoncom.dll and suggested a solution in bpo-7833.

First question: Open a new issue, or discuss it in this one (and repoen it)?

Please open a new issue. AFAICT, the original issues request (do not
embed manifests in pyd files) is now implemented fully. Feel free to
leave a link to the new issue here, but please start over describing the
problem you are seeing, and in what version you are seeing it.

Navigation Menu

Repository navigation

Do not embed manifest files in *.pyd when compiling with MSVC #48370

31 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Do not embed manifest files in *.pyd when compiling with MSVC #48370

Description

Activity

aheider commented on Oct 14, 2008

loewis commented on Oct 14, 2008

aheider commented on Oct 14, 2008

loewis commented on Oct 14, 2008

aheider commented on Oct 14, 2008

loewis commented on Oct 14, 2008

aheider commented on Oct 16, 2008

loewis commented on Oct 17, 2008

koen commented on Oct 17, 2008

aheider commented on Oct 20, 2008

koen commented on Oct 20, 2008

31 remaining items

cgohlke commented on Oct 18, 2009

cgohlke commented on Oct 29, 2009

malemburg commented on Oct 29, 2009

cgohlke commented on Oct 29, 2009

koen commented on Oct 29, 2009

egaudry commented on Dec 1, 2009

loewis commented on Dec 3, 2009

skrah commented on Dec 19, 2009

srid commented on Dec 21, 2009

theller commented on Apr 26, 2010

egaudry commented on Apr 26, 2010

cgohlke commented on Apr 26, 2010

loewis commented on Apr 26, 2010

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions