Repository navigation
bdist_msi generates version number for pure Python packages #49561
Description
| BPO | 5311 |
|---|---|
| Nosy | @loewis, @tarekziade |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2009-05-05.02:23:30.015>
created_at = <Date 2009-02-19.04:16:05.848>
labels = ['type-feature', 'library']
title = 'bdist_msi generates version number for pure Python packages'
updated_at = <Date 2009-05-05.02:23:30.014>
user = 'https://bugs.python.org/bethard'bugs.python.org fields:
activity = <Date 2009-05-05.02:23:30.014>
actor = 'bethard'
assignee = 'bethard'
closed = True
closed_date = <Date 2009-05-05.02:23:30.015>
closer = 'bethard'
components = ['Distutils']
creation = <Date 2009-02-19.04:16:05.848>
creator = 'bethard'
dependencies = []
files = ['13823', '13841', '13855', '13856']
hgrepos = []
issue_num = 5311
keywords = ['patch']
message_count = 26.0
messages = ['82455', '82459', '82460', '82461', '82462', '84760', '84792', '84994', '85002', '85760', '86853', '86854', '86855', '86965', '86980', '86992', '87060', '87062', '87065', '87066', '87136', '87146', '87149', '87157', '87164', '87202']
nosy_count = 4.0
nosy_names = ['loewis', 'atuining', 'bethard', 'tarek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5311'
versions = ['Python 3.1', 'Python 2.7']
Activity
bethard commentedon Feb 19, 2009
I just ran "setup.py bdist_msi" with NLTK which is a pure Python
package. You can see the setup.py here:
http://code.google.com/p/nltk/source/browse/trunk/nltk/setup.py. Despite
the fact that NLTK is pure Python, the generated .msi's look like
nltk-0.9.8.win32-py2.5.msi and nltk-0.9.8.win32-py2.6.msi (built with
Python 2.5 and 2.6 respectively).
So, two questions: (1) are the version numbers supposed to be there? and
(2) if so, does that mean a .msi for a pure Python package built by
Python 2.6 won't work on any other version?
loewis commentedon Feb 19, 2009
Yes.
Yes.
bethard commentedon Feb 19, 2009
Mostly out of curiosity, why is that? With bdist_wininst, a pure Python
package would generate a version-less installer that could then be used
with any Python version.
loewis commentedon Feb 19, 2009
Primarily because it's not implemented. To implement it, you
would need to collect all Python installations on the system
from the registry, then create a UI to let the user select
a specific installation, then use that. Collecting all Python
versions is fairly difficult to do with standard MSI actions.
In addition, a common use case is that MSI installation works
unattended (no UI), in which case you would also have to make
a choice of default version to install to (e.g. with highest
version number).
Contributions are welcome.
bethard commentedon Feb 19, 2009
I'm certainly no Windows API expert, but if no one takes a stab at it
sooner, maybe I can spend some time looking at this during PyCon.
I'm switching the ticket type to a feature request.
bethard commentedon Mar 31, 2009
Ok, so here's what needs to happen to make this work. Note that all of
the following needs to happen at *runtime*, not at the time at which the
.msi is created:
(1) Find all sub-keys of SOFTWARE\Python\PythonCore in the registry.
These are the versions, e.g. 2.5, 2.6, 3.0, etc.
(2) For each version, get the Python installation dir from
SOFTWARE\Python\PythonCore\%(version)s\InstallPath
(3) Populate the ListView table with entries containing these
installation paths, e.g. something like:
(4) Modify Control/SelectDirectoryDlg so that it uses a ListView (filled
with the above values) instead of a DirectoryCombo.
(5) Make a couple minor edits to bdist_msi.py to stop it from inserting
the version into the .msi name for Python-only modules.
I looked into a couple ways of doing this. Ideally, we should avoid
using a CustomAction, which would require maintaining some additional C
or VBScript code, and instead do everything through the database tables
that are built into all .msi files. Some problems I've run into with
this approach:
The only way to read the registry AFAICT is through the RegLocator
table: http://msdn.microsoft.com/en-us/library/aa371171(VS.85).aspx. But
RegLocator can only look up single key values, and cannot look up the
sub-keys under a key (which is what we need to get the Python versions).
We could hard code all the possible versions, and stick all the
corresponding SOFTWARE\...\%(version)s\InstallPath keys into RegLocator.
Then these can be read into properties using the AppSearch table
(http://msdn.microsoft.com/en-us/library/aa371559(VS.85).aspx), and we
could then fill in the ListView table with the install paths. But
AFAICT, there is no way to keep from putting one row into the ListView
table for each version of Python we statically define. Which means a
bunch of inappropriate rows at runtime (e.g. there'd be a row for Python
2.3 even if there was no Python 2.3 on your system).
Basically, the problem is that we'd like to determine what goes into the
ListView table at runtime, but I can only figure out how to put things
into it at the time at which the .msi is built.
I'm going to continue to look into this, and I'd welcome suggestions
anyone has.
The last resort would be to create a CustomAction using a DLL or
VBScript, but I'm really trying to avoid that, both because maintaining
and debugging such code is painful, and because Michael Foord suggested
that some virus checkers will complain about .msi files with embedded
VBScript (probably forcing me to maintain a DLL - ugh!).
loewis commentedon Mar 31, 2009
In
http://www.installsite.org/pages/en/msi/articles/MultiListBox/index.htm
there is a demo how to modify the listbox contents dynamically using
VBScript.
bethard commentedon Apr 1, 2009
Thanks for the link. I did play around with that code for quite a while,
and while I'm convinced there's a way to get it to work, I'm still
struggling with it. I believe the attached .vbs file does about what we
need to do, but I think I'm attaching it at the wrong location in the
.msi file because things seem to die at the "Session.Database.OpenView"
line with an error saying the object doesn't have that method (or no
error message but still at that line, depending on how I've attached the
action.)
Anyway, I'll probably keep playing around with this, but some food for
thought on why we may not want to implement this as a .vbs:
http://blogs.msdn.com/robmen/archive/2004/05/20/136530.aspx
In particular, some anti-virus products will silently keep them from
working. That said, I probably need to figure out how to write the .vbs
code in order to (eventually) write the C or whatever else code.
bethard commentedon Apr 1, 2009
My OpenView bug was just a missing "Set". The CustomAction does seem to
be correctly gathering the Python paths from the registry and filling
the ListView now. I've still got a couple of errors showing up later in
the process, but I expect I'll probably have a patch within a week or two.
5 remaining items
bethard commentedon May 2, 2009
Ok, I've been chatting with folks on microsoft.public.platformsdk.msi,
and I think the right approach here is to define a Feature for each
version of Python. Each Feature would install the exact same files, but
to a different Python directory.
One of the nice things about this approach would be that you could
install a module for multiple Python versions all at the same time. It
also should avoid the need for a VBScript or C-based CustomAction - we
should be able to do everything with the database tables.
One of the downsides is that we'll have to hard-code in all possible
versions of Python. My current plan is just to declare all versions as
2.0, 2.1, ..., 3.9. That should be good enough for a while, but at some
point this will have to get updated to work with Python 4.0. ;-)
I'll post back here when I've made some progress on this.
bethard commentedon May 2, 2009
Ok, that was actually easier than I thought it would be. The new patch
introduces properties for each Python version (e.g. TARGETDIR2.4,
PYTHON.MACHINE.2.4, etc.), and disables and hides the features for any
Python versions that aren't found in the registry.
The one remaining issue: What should we do about Python installations
that are missing the appropriate keys in the registry? I imagine this
could happen if, say, you build Python from source. My first thought was
to add an "Other Python Installation" Feature that is disabled (but
visible) by default, and allow the path for that Feature to be filled in
by hand. Does that make sense?
bethard commentedon May 3, 2009
Ok, I added one final Feature that allows the user to specify an
alternate Python directory. (The PathEdit for specifying the directory
will only display if this Feature is set to be installed.)
I think this patch is pretty much ready to go in now. It could use a
review and some testing from folks on other kinds of machines, but I
think all the functionality is there now. (And no need for CustomAction
scripts! Yay!)
If at all possible, I'd like to get this into the Python 3.1 beta so
that we can have as many people as possible test this out on other
Windows machines.
bethard commentedon May 3, 2009
A slightly improved patch, using DuplicateFile instead of storing a copy
of each file for each Python version. Should keep the size of the
resulting MSI similar to the size of the currently generated MSIs.
bethard commentedon May 3, 2009
Ok, one last tiny update that makes sure TARGETDIR is always set to one
of the TARGETDIRX.Ys from a Feature that is actually selected.
I swear I'm done with this now. ;-)
loewis commentedon May 3, 2009
Can you kindly attach a demo MSI, to simplify review?
bethard commentedon May 3, 2009
Here's an MSI generated for the argparse module.
loewis commentedon May 4, 2009
The patch looks fine so far, please apply to trunk and 3k. As this is a
new feature, I think backporting it is not appropriate.
I believe that the support for (pre)install scripts is incorrect; I
would expect that each such script should be executed once per version
for which the package is installed. However, this can be fixed later;
for the moment, it would be good enough to warn at packaging time that
scripts may not work for multi-version packages.
You can also consider providing "Change" installation, allowing the user
to selectively add a package to a python version after that python
version is installed (or to remove it before the python version gets
removed). To do so, just add an option to the MaintenanceTypeDlg that
forks into the feature selection dialog, with the standard INSTALL action.
atuining commentedon May 4, 2009
One additional suggestion: allow the packager to specify what the
minimum Python version is. Otherwise, you might have a package that
enables installation for Python 2.3 and 2.4 when the maintainer has
already stated that Python 2.5 is the minimum version supported.
Otherwise, looks great.
bethard commentedon May 4, 2009
@martin:
Thanks! Do I need to do something special to make the merging work
right? Or do I just apply the patch separately to the trunk and the py3k
branche?
Good point about the install script - I think the condition needs to be
("&Python%s=3" % ver) instead of "NOT Installed". I'll use this approach
when I apply the patch, but clearly this needs some testing by folks
that use install scripts either way. I didn't worry much about the
pre-install script, since finalize_options() already throws an exception
saying that isn't supported
I'll open a new issue for support for a "Change" installation. This is
one of the nice things that treating Python versions as features allows,
I just haven't had time to look into it yet.
@anthony:
Allowing a minimum version to be set sounds like a nice feature. Could
you open a new feature request?
atuining commentedon May 4, 2009
I've created another feature request as requested for supplying a
minimum Python version when creating pure Python packages.
http://bugs.python.org/issue5926
loewis commentedon May 4, 2009
You commit to the trunk, then you do "svnmerge merge -r<rev>" in the 3k
branch, then "svn commit -F svnmerge-something.txt" (in case of
conflicts, you fix them first, of course).
I'm not sure - I think the install script must run several times
actually, so there must be several custom actions, each with its own
condition.
Well, if there had been separate packages per version, you could install
them independently, anyway.
bethard commentedon May 5, 2009
Done in r72306 for trunk and r72309 for 3k. Thanks.
Yep. That's what the patch does - adds a CustomAction and an entry in
InstallExecuteSequence for each version.
For what it's worth, you still have this option if anyone wants to
distribute multiple MSIs. Simply pass --target-version to bdist_msi, and
you'll get an MSI for only the selected version of Python that you can
install independently of other MSIs generated with --target-version.