Difficulties building tip of tree Python for emscripten

@tiran

BPO	47232
Nosy	@tiran, @pmp-p, @hoodmane
Files	Makefile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-04-05.22:33:35.926>
created_at = <Date 2022-04-05.19:27:14.891>
labels = []
title = 'Difficulties building tip of tree Python for emscripten'
updated_at = <Date 2022-04-05.22:33:35.925>
user = 'https://github.com/hoodmane'

bugs.python.org fields:

activity = <Date 2022-04-05.22:33:35.925>
actor = 'hoodchatham'
assignee = 'none'
closed = True
closed_date = <Date 2022-04-05.22:33:35.926>
closer = 'hoodchatham'
components = []
creation = <Date 2022-04-05.19:27:14.891>
creator = 'hoodchatham'
dependencies = []
files = ['50723']
hgrepos = []
issue_num = 47232
keywords = []
message_count = 8.0
messages = ['416807', '416812', '416815', '416816', '416819', '416820', '416821', '416824']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'hoodchatham']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue47232'
versions = []

I am trying to build tot Python for Emscripten to test recent changes and having trouble. The problem is caused by recent changes since 3.11.0a6 to sre. I build a build Python 3.11 from tot, and provided it to ./configure --with-build-python=$(HOSTBUILDDIR)/python. However, I still see invocations to the system Python 3.11 at /usr/local/bin/python3.11.

The problem seems to be that the generated Makefile has a bunch of stuff collapsed into one line:

PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) /src/cpython/build/Python-3.11.0dev0/builddir/build-host/python

manually fixing this to:

PYTHON_FOR_BUILD=/src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) 
PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)

fixes the problem.

My configure invocation is as follows:

	( \
		cd $(BUILDDIR); \
		CONFIG_SITE=./config.site-wasm32-emscripten emconfigure \
		  ../../configure \
		  	  --cache-file=$(ROOT)/python.config.cache \
			  CFLAGS="${PYTHON_CFLAGS}" \
			  CPPFLAGS="-I$(SQLITEBUILD) -I$(BZIP2BUILD) -I$(ZLIBBUILD)" \
			  PLATFORM_TRIPLET="$(PLATFORM_TRIPLET)" \
			  --with-build-python=$(HOSTBUILDDIR)/python \
			  --without-pymalloc \
			  --disable-shared \
			  --disable-ipv6 \
			  --enable-big-digits=30 \
			  --enable-optimizations \
			  --host=wasm32-unknown-emscripten\
			  --build=$(shell $(EXTRACTDIR)/config.guess) \
			  --prefix=$(INSTALL)  \
	)

Okay I found the problem. Elsewhere in my Makefiles PYTHON_FOR_BUILD was getting overwritten.

I take it back, I am still having this problem.

Please attach config.log and Makefile from the builddir.

config.log is apparently 1.7 MB and when I try to upload I get "Error 413: Entity Too Large". I've attached the Makefile.

There is still a pretty good change I am making some dumb mistake.

You are building a cross-build with "--with-build-python=/usr/local/bin/python3.11". Is the interpreter up to date?

During alpha and beta phase, the build Python interpreter should be built from the exact same git commit as the sources in "../..". If the interpreter deviates from the sources in your srcdir, then Python byte code (.pyc), frozen byte code, and re engine can have mismatching magic.

I recommend that you build inside an environment that does not have Python 3.11 installed. Instead build a build Python interpreter from the same checkout that you later use for cross-compilation. "./configure && make" is sufficient. You don't have to install the build interpreter. "--with-build-python=$(pwd)/../build/python" works fine.

We use this approach in python-wasm:

mkdir -p cpython/builddir/build
pushd cpython/builddir/build
../../configure -C
make -j$(nproc)
popd

mkdir -p cpython/builddir/emscripten-browser
pushd cpython/builddir/emscripten-browser
CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
emconfigure ../../configure -C \
--host=wasm32-unknown-emscripten \
--build=$(../../config.guess) \
--with-build-python=$(pwd)/../build/python
emmake make -j$(nproc)

It's going to get easier and less painful as soon as we reach beta phase. During beta the byte code will only change when it is required for a bug fix.

Okay I got it working. I was making several mistakes that each caused the same symptoms so it was hard to track them all down.

Thanks again for your help!

Navigation Menu

Repository navigation

Difficulties building tip of tree Python for emscripten #91388

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Difficulties building tip of tree Python for emscripten #91388

Description

Activity

hoodmane commented on Apr 5, 2022

hoodmane commented on Apr 5, 2022

hoodmane commented on Apr 5, 2022

tiran commented on Apr 5, 2022

hoodmane commented on Apr 5, 2022

hoodmane commented on Apr 5, 2022

tiran commented on Apr 5, 2022

hoodmane commented on Apr 5, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions