Repository navigation
[2.7] bpo-30450: Pull Windows dependencies from GitHub rather than SVN (GH-1783) #3306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
8bff83d
bpo-30450: Pull Windows dependencies from GitHub rather than svn (GH-…
zware 3d6896b
Rename db -> bsddb for disambiguity
zware 3e9dc95
Update sqlite3 to 3.14.2.0
zware 91f23be
bpo-30687: Fixes build scripts to find msbuild.exe and stop relying o…
zooba f4072f7
Add PCbuild/*.ilk to .gitignore
zware 1441663
Cherry-pick parts of 68d663cf85d1ac5eaf83482eed39c0a6f8093601
zooba 6890ac3
bpo-30450: Fall back to git.exe if no Python is found. (#2739)
zooba 3db3ecd
Add NEWS blurb
zware 90f361c
Fixes doc/make.bat to properly handle quoted paths.
zooba e3c9556
bpo-30450: Fix logic for retrying nuget.exe download (#2744)
zooba File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
Misc/NEWS.d/next/Windows/2017-09-04-14-00-37.bpo-30450.YwitaJ.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Pull build dependencies from GitHub rather than svn.python.org. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| @rem | ||
| @rem Searches for MSBuild.exe. This is the only tool we need to initiate | ||
| @rem a build, so we no longer search for the full VC toolset. | ||
| @rem | ||
| @rem This file is supposed to modify the state of the caller (specifically | ||
| @rem the MSBUILD variable), so we do not use setlocal or echo, and avoid | ||
| @rem changing any other persistent state. | ||
| @rem | ||
|
|
||
| @rem No arguments provided means do full search | ||
| @if '%1' EQU '' goto :begin_search | ||
|
|
||
| @rem One argument may be the full path. Use a goto so we don't try to | ||
| @rem parse the next if statement - incorrect quoting in the multi-arg | ||
| @rem case can cause us to break immediately. | ||
| @if '%2' EQU '' goto :one_arg | ||
|
|
||
| @rem Entire command line may represent the full path if quoting failed. | ||
| @if exist "%*" (set MSBUILD="%*") & (set _Py_MSBuild_Source=environment) & goto :found | ||
| @goto :begin_search | ||
|
|
||
| :one_arg | ||
| @if exist "%~1" (set MSBUILD="%~1") & (set _Py_MSBuild_Source=environment) & goto :found | ||
|
|
||
| :begin_search | ||
| @set MSBUILD= | ||
|
|
||
| @rem If msbuild.exe is on the PATH, assume that the user wants that one. | ||
| @where msbuild > "%TEMP%\msbuild.loc" 2> nul && set /P MSBUILD= < "%TEMP%\msbuild.loc" & del "%TEMP%\msbuild.loc" | ||
| @if exist "%MSBUILD%" set MSBUILD="%MSBUILD%" & (set _Py_MSBuild_Source=PATH) & goto :found | ||
|
|
||
| @rem VS 2015 and earlier register MSBuild separately, so we can find it. | ||
| @rem Prefer MSBuild 14.0 over MSBuild 15.0, since the latter may not be able to find a VC14 install. | ||
| @reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\14.0" /v MSBuildToolsPath /reg:32 >nul 2>nul | ||
| @if NOT ERRORLEVEL 1 @for /F "tokens=1,2*" %%i in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\14.0" /v MSBuildToolsPath /reg:32') DO @( | ||
| @if "%%i"=="MSBuildToolsPath" @if exist "%%k\msbuild.exe" @(set MSBUILD="%%k\msbuild.exe") | ||
| ) | ||
| @if exist %MSBUILD% (set _Py_MSBuild_Source=registry) & goto :found | ||
|
|
||
| @rem VS 2017 sets exactly one install as the "main" install, so we may find MSBuild in there. | ||
| @reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7" /v 15.0 /reg:32 >nul 2>nul | ||
| @if NOT ERRORLEVEL 1 @for /F "tokens=1,2*" %%i in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7" /v 15.0 /reg:32') DO @( | ||
| @if "%%i"=="15.0" @if exist "%%k\MSBuild\15.0\Bin\msbuild.exe" @(set MSBUILD="%%k\MSBuild\15.0\Bin\msbuild.exe") | ||
| ) | ||
| @if exist %MSBUILD% (set _Py_MSBuild_Source=Visual Studio 2017 registry) & goto :found | ||
|
|
||
|
|
||
| @exit /b 1 | ||
|
|
||
| :found | ||
| @echo Using %MSBUILD% (found in the %_Py_MSBuild_Source%) | ||
| @set _Py_MSBuild_Source= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| @rem | ||
| @rem Searches for python.exe and may download a private copy from nuget. | ||
| @rem | ||
| @rem This file is supposed to modify the state of the caller (specifically | ||
| @rem the MSBUILD variable), so we do not use setlocal or echo, and avoid | ||
| @rem changing any other persistent state. | ||
| @rem | ||
|
|
||
| @rem No arguments provided means do full search | ||
| @if '%1' EQU '' goto :begin_search | ||
|
|
||
| @rem One argument may be the full path. Use a goto so we don't try to | ||
| @rem parse the next if statement - incorrect quoting in the multi-arg | ||
| @rem case can cause us to break immediately. | ||
| @if '%2' EQU '' goto :one_arg | ||
|
|
||
| @rem Entire command line may represent the full path if quoting failed. | ||
| @if exist "%*" (set PYTHON="%*") & (set _Py_Python_Source=from environment) & goto :found | ||
| @goto :begin_search | ||
|
|
||
| :one_arg | ||
| @if exist "%~1" (set PYTHON="%~1") & (set _Py_Python_Source=from environment) & goto :found | ||
|
|
||
| :begin_search | ||
| @set PYTHON= | ||
|
|
||
| @set _Py_EXTERNALS_DIR=%EXTERNAL_DIR% | ||
| @if "%_Py_EXTERNALS_DIR%"=="" (set _Py_EXTERNALS_DIR=%~dp0\..\externals) | ||
|
|
||
| @rem If we have Python in externals, use that one | ||
| @if exist "%_Py_EXTERNALS_DIR%\pythonx86\tools\python.exe" (set PYTHON="%_Py_EXTERNALS_DIR%\pythonx86\tools\python.exe") & (set _Py_Python_Source=found in externals directory) & goto :found | ||
|
|
||
| @rem If py.exe finds a recent enough version, use that one | ||
| @py -3.6 -V >nul 2>&1 && (set PYTHON=py -3.6) && (set _Py_Python_Source=found with py.exe) && goto :found | ||
|
|
||
| @if NOT exist "%_Py_EXTERNALS_DIR%" mkdir "%_Py_EXTERNALS_DIR%" | ||
| @set _Py_NUGET=%NUGET% | ||
| @set _Py_NUGET_URL=%NUGET_URL% | ||
| @if "%_Py_NUGET%"=="" (set _Py_NUGET=%EXTERNALS_DIR%\nuget.exe) | ||
| @if "%_Py_NUGET_URL%"=="" (set _Py_NUGET_URL=https://aka.ms/nugetclidl) | ||
| @if NOT exist "%_Py_NUGET%" ( | ||
| @echo Downloading nuget... | ||
| @rem NB: Must use single quotes around NUGET here, NOT double! | ||
| @rem Otherwise, a space in the path would break things | ||
| @powershell.exe -Command Invoke-WebRequest %_Py_NUGET_URL% -OutFile '%_Py_NUGET%' | ||
| ) | ||
| @echo Installing Python via nuget... | ||
| @"%_Py_NUGET%" install pythonx86 -ExcludeVersion -OutputDirectory "%_Py_EXTERNALS_DIR%" | ||
| @rem Quote it here; it's not quoted later because "py -3.6" wouldn't work | ||
| @if not errorlevel 1 (set PYTHON="%_Py_EXTERNALS_DIR%\pythonx86\tools\python.exe") & (set _Py_Python_Source=found on nuget.org) & goto :found | ||
|
|
||
|
|
||
| @exit /b 1 | ||
|
|
||
| :found | ||
| @echo Using %PYTHON% (%_Py_Python_Source%) | ||
| @set _Py_Python_Source= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| import argparse | ||
| import os | ||
| import pathlib | ||
| import zipfile | ||
| from urllib.request import urlretrieve | ||
|
|
||
|
|
||
| def fetch_zip(commit_hash, zip_dir, *, org='python', binary=False, verbose): | ||
| repo = f'cpython-{"bin" if binary else "source"}-deps' | ||
| url = f'https://github.com/{org}/{repo}/archive/{commit_hash}.zip' | ||
| reporthook = None | ||
| if verbose: | ||
| reporthook = print | ||
| zip_dir.mkdir(parents=True, exist_ok=True) | ||
| filename, headers = urlretrieve( | ||
| url, | ||
| zip_dir / f'{commit_hash}.zip', | ||
| reporthook=reporthook, | ||
| ) | ||
| return filename | ||
|
|
||
|
|
||
| def extract_zip(externals_dir, zip_path): | ||
| with zipfile.ZipFile(os.fspath(zip_path)) as zf: | ||
| zf.extractall(os.fspath(externals_dir)) | ||
| return externals_dir / zf.namelist()[0].split('/')[0] | ||
|
|
||
|
|
||
| def parse_args(): | ||
| p = argparse.ArgumentParser() | ||
| p.add_argument('-v', '--verbose', action='store_true') | ||
| p.add_argument('-b', '--binary', action='store_true', | ||
| help='Is the dependency in the binary repo?') | ||
| p.add_argument('-O', '--organization', | ||
| help='Organization owning the deps repos', default='python') | ||
| p.add_argument('-e', '--externals-dir', type=pathlib.Path, | ||
| help='Directory in which to store dependencies', | ||
| default=pathlib.Path(__file__).parent.parent / 'externals') | ||
| p.add_argument('tag', | ||
| help='tag of the dependency') | ||
| return p.parse_args() | ||
|
|
||
|
|
||
| def main(): | ||
| args = parse_args() | ||
| zip_path = fetch_zip( | ||
| args.tag, | ||
| args.externals_dir / 'zips', | ||
| org=args.organization, | ||
| binary=args.binary, | ||
| verbose=args.verbose, | ||
| ) | ||
| final_name = args.externals_dir / args.tag | ||
| extract_zip(args.externals_dir, zip_path).replace(final_name) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| main() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have an open PR making this bit even better! You should review that before backporting something different :)