Statically linked sqlite3 fails checks due to missing -lm

Bug report

Bug description:

On Linux, if I build a static version of sqlite3 (e.g., ./configure --disable-shared && make from a sqlite3 tarball)

You can see that ./configure for CPython fails the SQLite checks (added back in #30016):

$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no

And in the config.log there are a bunch of failures due to undefined references to functions in libm, e.g.:

configure:15469: checking for sqlite3_bind_double in -lsqlite3
configure:15492: gcc -o conftest   -I/usr/local/include  -I$(srcdir)/Modules/_sqlite -L/usr/local/lib -lsqlite3  -L/usr/local/lib conftest.c -lsqlite3  -L/usr/local/lib -lsqlite3  -ldl  >&5
/usr/bin/ld: /usr/local/lib/libsqlite3.a(sqlite3.o): in function `logFunc':
/repo/sqlite-autoconf-3470100/sqlite3.c:131765:(.text+0x26858): undefined reference to `log2'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131768:(.text+0x268d4): undefined reference to `log'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131762:(.text+0x268ec): undefined reference to `log10'
/usr/bin/ld: /repo/sqlite-autoconf-3470100/sqlite3.c:131750:(.text+0x2691c): undefined reference to `log'

CPython is determing SQLite's requirements via pkg-config if it can at:

cpython/configure.ac

Lines 4204 to 4209 in 492b224

    
           PKG_CHECK_MODULES( 
        
             [LIBSQLITE3], [sqlite3 >= 3.15.2], [], [ 
        
               LIBSQLITE3_CFLAGS=${LIBSQLITE3_CFLAGS-""} 
        
               LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"} 
        
             ] 
        
           )

And -lm is included in the pkg-config for the static sqlite3 build, but it's in the private libs and requires the --static flag to be included in the --libs output:

$ pkg-config --static --libs sqlite3 
-L/usr/local/lib -lsqlite3 -lm 
$ pkg-config --libs sqlite3             
-L/usr/local/lib -lsqlite3

However, PKG_CHECK_MODULES doesn't pass this flag. There's a PKG_CHECK_MODULES_STATIC macro which does, but CPython doesn't seem to use this yet. The alternative is set the flag manually at configure time:

PKG_CONFIG="$PKG_CONFIG --static"

which changes all the PKG_CHECK_MODULES calls to include the --static flag. However, there's a missing export for LIBS which prevents this from resolving the issue:

$ git rev-parse HEAD
f9a5a3a3ef34e63dc197156e9a5f57842859ca04
$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
checking for --enable-loadable-sqlite-extensions... no
checking for stdlib extension module _sqlite3... missing
^C

This is "solved" with the patch:

diff --git a/configure.ac b/configure.ac
index badb19d5589..5ed147f5482 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4221,6 +4221,7 @@ dnl bpo-45774/GH-29507: The CPP check in AC_CHECK_HEADER can fail on FreeBSD,
 dnl hence CPPFLAGS instead of CFLAGS.
   CPPFLAGS="$CPPFLAGS $LIBSQLITE3_CFLAGS"
   LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS"
+  LIBS="$LIBSQLITE3_LIBS $LIBS"
 
   AC_CHECK_HEADER([sqlite3.h], [
     have_sqlite3=yes

As demonstrated by:

$ export PKG_CONFIG="pkg-config --static"
$ ./configure 2>&1 | grep sqlite
checking for sqlite3 >= 3.15.2... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... yes
checking for sqlite3_column_decltype in -lsqlite3... yes
checking for sqlite3_column_double in -lsqlite3... yes
checking for sqlite3_complete in -lsqlite3... yes
checking for sqlite3_progress_handler in -lsqlite3... yes
checking for sqlite3_result_double in -lsqlite3... yes
checking for sqlite3_set_authorizer in -lsqlite3... yes
checking for sqlite3_trace_v2 in -lsqlite3... yes
checking for sqlite3_value_double in -lsqlite3... yes
checking for sqlite3_load_extension in -lsqlite3... yes
checking for sqlite3_serialize in -lsqlite3... yes
checking for --enable-loadable-sqlite-extensions... no

For context, this was reported in astral-sh/python-build-standalone#449 and some of the findings here are repeated there.

I've posted the patch in #128322

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

@erlend-aasland

cc @erlend-aasland — I understand you're the expert in this area.

In addition to #128322, we may also want to include -lm at

cpython/configure.ac

Line 4207 in 492b224

LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"}

If the pkg-config check fails, we fallback to including these flags. However, without -lm all of our subsequent checks will fail! Happy to post a pull request for that too.

In addition to #128322, we may also want to include -lm at

cpython/configure.ac

Line 4207 in 492b224

LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"}

If the pkg-config check fails, we fallback to including these flags. However, without -lm all of our subsequent checks will fail! Happy to post a pull request for that too.

This will result in over-linking when sqlite3 is used as a shared library (no pkg-config --static).

This will result in over-linking when sqlite3 is used as a shared library

Ah true — we could just include it during the checks but it seems like a bit of a pain.

gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 requirem…

pythongh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 re…

[3.12] gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 r…

[3.13] gh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 r…

pythongh-128321: Set LIBS instead of LDFLAGS when checking sqlite3 re…

@zanieb

@zanieb Thank you for finding and fixing this. It's a huge help.

This will result in over-linking when sqlite3 is used as a shared library

Ah true — we could just include it during the checks but it seems like a bit of a pain.

I'm probably missing something, and if so please forgive me.

Since cpython won't be using pkgconfig --static any time soon, is it therefor impossible for anyone to compile sqlite3 in statically without patching configure themselves just to append -lm to the LIBS, and just to successfully satisfy the check? If so, does the overlinking represent a risk for the sqlite symbol check that is worth leaving it in this state, which is basically just broken for this use case?

Typically what happens is people who are interested in a static build export PKG_CONFIG="pkg-config --static" to ensure that all dependencies are detected statically. That will include -lm and also -lz (the latter wasn't mentioned here and it is needed if you build a dynamic python that is statically linked to the sqlite library).

So everything works fine using pkg-config, the question is what to do about the fallback.

For a fallback, what you'd typically do is perform a series of compile checks, to see if you can link successfully to sqlite on its own and upon failure check again to see if you can link successfully to sqlite when adding -lm -lz.

I have no objections to the "series of compile checks". I don't think it's fully appropriate in terms of general theory to simply add it unconditionally though. I'm also fine with only supporting pkg-config at all, for my own personal use, but that's another story. :)

I see. It seems to me the project is designed to support static/dynamic choices (and all configuration choices for that matter), on a per-dependency basis. I definitely see it being "very common" that people who want "some deps" compiled in statically want "all deps" statically. On the other hand, it's also "not uncommon" for people to want a subset of deps static. Regardless, it doesn't seem appropriate even in principle to have an "all static or nothing static" support story, just because pkgconfig makes the "all static" case easy. So many other configurations and mechanisms in cpython are in place to make it flexible in this way.

Navigation Menu

Repository navigation

Statically linked `sqlite3` fails checks due to missing `-lm` #128321

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	PKG_CHECK_MODULES(
	[LIBSQLITE3], [sqlite3 >= 3.15.2], [], [
	LIBSQLITE3_CFLAGS=${LIBSQLITE3_CFLAGS-""}
	LIBSQLITE3_LIBS=${LIBSQLITE3_LIBS-"-lsqlite3"}
	]
	)

Statically linked sqlite3 fails checks due to missing -lm #128321

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

zanieb commented on Dec 29, 2024

zanieb commented on Dec 29, 2024

eli-schwartz commented on Dec 29, 2024

zanieb commented on Dec 30, 2024

solvingj commented on Feb 12, 2025

eli-schwartz commented on Feb 12, 2025

solvingj commented on Feb 12, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Statically linked `sqlite3` fails checks due to missing `-lm` #128321