Error to build _dbm module during make

@smontanaro

BPO	4483
Nosy	@smontanaro
Files	dbm.diff: Possible Ubuntu/Lenny fix dbm2.diff: Python 2.x patch for Ubuntu/Lenny dbm2.diff dbm3.diff: Patch allowing Python 3 to build on Debian Lenny dbm.diff dbm.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/smontanaro'
closed_at = <Date 2008-12-06.17:53:34.212>
created_at = <Date 2008-12-01.22:35:11.210>
labels = ['extension-modules', 'easy', 'build']
title = 'Error to build _dbm module during make'
updated_at = <Date 2013-02-08.16:04:34.118>
user = 'https://bugs.python.org/legerf'

bugs.python.org fields:

activity = <Date 2013-02-08.16:04:34.118>
actor = 'ddvento@ucar.edu'
assignee = 'skip.montanaro'
closed = True
closed_date = <Date 2008-12-06.17:53:34.212>
closer = 'skip.montanaro'
components = ['Extension Modules']
creation = <Date 2008-12-01.22:35:11.210>
creator = 'legerf'
dependencies = []
files = ['12181', '12182', '12202', '12230', '12231', '12234']
hgrepos = []
issue_num = 4483
keywords = ['patch', 'easy']
message_count = 30.0
messages = ['76709', '76720', '76721', '76722', '76760', '76761', '76763', '76766', '76769', '76770', '76873', '76937', '76948', '76949', '76951', '76956', '76958', '76962', '77074', '77122', '77154', '77168', '77169', '77205', '77256', '77269', '77275', '77279', '77282', '181679']
nosy_count = 7.0
nosy_names = ['skip.montanaro', 'flub', 'tlesher', 'rpetrov', 'legerf', 'AndyP', 'ddvento@ucar.edu']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue4483'
versions = ['Python 3.0', 'Python 2.7']

Under Debian/Lenny/i486, I install python3.0rc3.
After untar the Python3.0rc3 package, I replace in the configure
"/usr/local" by "/usr". After I do :
"configure --with-pydebug --with-doc-strings --enable-shared
--enable-profiling --enable-ipv6 --with-threads --with-tsc ; make"

The result during make is :
"Failed to find the necessary bits to build these modules:
_dbm
To find the necessary bits, look in setup.py in detect_modules() for the
module's name."

The release of the dbm library is :
libgdbm-dev 1.8.3-3 GNU dbm database routines (development files)

The header for dbm : "_gdbmmodule.c" request "gdbm.h", and is here
"/usr/include/gdbm.h"

The "Modules/_dbmmodule.c" use :
#include <gdbm/ndbm.h>" ... #include <ndbm.h> ...
but I have only "/usr/include/gdbm-ndbm.h"

Can you patch the "_dbmmodule.c" to use it ?

Patching _dbmmodule.c alone isn't sufficient. At minimum setup.py would
have to be adjusted to detect the presence of the odd ndbm.h file.

Can you try the attached patch and let us know if it works for you?

Here's a similarly untested patch for Python 2.x. Can you try it as well?

I modify your dbm2.diff because the module name is "_dbmmodule.c" and
not "dbmmodule.c".
Patching success
new test "configure ... ; make" and make stop with error :
"...
gcc -pthread -shared
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/_ctypes.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/callbacks.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/callproc.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/stgdict.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/cfield.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/malloc_closure.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/prep_cif.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/x86/ffi.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/x86/sysv.o
-L/usr/local/lib -L. -lpython3.0 -o
build/lib.linux-i686-3.0-pydebug/_ctypes.so

Failed to build these modules:
_dbm

running build_scripts
creating build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/pydoc ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/idle ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/2to3 ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Lib/smtpd.py ->
build/scripts-3.0
changing mode of build/scripts-3.0/pydoc from 644 to 755
changing mode of build/scripts-3.0/idle from 644 to 755
changing mode of build/scripts-3.0/2to3 from 644 to 755
changing mode of build/scripts-3.0/smtpd.py from 644 to 755
[88798 refs]"

I retest with light configure modification and dbm2.diff with file name
"_dbmmodule.c" :

"configure --with-pydebug --with-doc-strings --enable-shared
--enable-profiling --enable-ipv6 --with-threads --with-tsc --prefix=/usr
; make"

"...
gcc -pthread -shared
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_dbmmodule.o
-L/usr/local/lib -L. -lgdbm -lpython3.0 -o
build/lib.linux-i686-3.0-pydebug/_dbm.so
*** WARNING: renaming "_dbm" since importing it failed:
build/lib.linux-i686-3.0-pydebug/_dbm.so: undefined symbol: dbm_firstkey

Failed to build these modules:
_dbm

running build_scripts
[89618 refs]"

Leger> - I modify your dbm2.diff because the module name is
Leger> "_dbmmodule.c" and not "dbmmodule.c".

I think you missed the point of the dbm2.diff file. It should be applied to
the Python 2.x code, not to Python 3.x. I don't have access to the
necessary platform to run the test.

To summarize:

* dbm.diff should be applied to the py3k branch (or to the 3.0rc3 release
  code).
* dbm2.diff should be applied to the code on the trunk (or to the 2.6
  release code).

Skip

Skip, Sorry for my mistake with dbm2.diff ;)

From the tar 3.0rc3 I apply the patch (dbm.diff). Configure ... and
make. But the result is always wrong :
"Failed to build these modules:
_dbm"

Best regards, Frederic

Frederic> From the tar 3.0rc3 I apply the patch (dbm.diff). Configure
Frederic> ... and make. But the result is always wrong :

> "Failed to build these modules:
> _dbm"

Why does it fail? Is there compile error? Does linking fail? I guessed
that the library would still be "gdbm", but I don't really know. Does your
system maybe install a library file named "libgdbm-ndbm.so" or something
similarly obscure?

I'm happy to try and get this resolved, but not that there is only another
day or two before 3.0final is released. (This might not make the release
anyway.)

Only the make generate last error sentence.
Debian/Lenny provide only "libgdbm-dev-1.8.3-3" provide :
root@xxx:/install# ll /usr/lib/libgdbm.so*
lrwxrwxrwx 1 root root 16 2008-12-01 21:45 /usr/lib/libgdbm.so ->
libgdbm.so.3.0.0
lrwxrwxrwx 1 root root 16 2008-03-08 21:38 /usr/lib/libgdbm.so.3 ->
libgdbm.so.3.0.0
-rw-r--r-- 1 root root 19940 2006-04-24 05:25 /usr/lib/libgdbm.so.3.0.0

I install the Python3.0 (final), the test make is the same and the
result too. I join the output.

Skip, your simplified patch works for me. Makes it build fine with the
following snippet:

building '_dbm' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_GDBM_DASH_NDBM_H -I.
-I/home/andy/src/python3/Python-3.0/./Include -I. -IInclude -I../Include
-I/usr/local/include -I/home/andy/src/python3/Python-3.0/debug/Include
-I/home/andy/src/python3/Python-3.0/debug -c
/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.c -o
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
gcc -pthread -shared
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
-L/usr/local/lib -lgdbm -lgdbm_compat -o build/lib.linux-i686-3.0/_dbm.so

Thanks.

This patch resolve the problem. Evidence with "make test" :
...
test_dbm
test_dbm_dumb
test_dbm_gnu
test_dbm_ndbm
...

Many thanks Skip. You can close this bug I think.

Hi, I'd like to confirm that Skip's last patch fixes the issue. Hope it
gets included soon!

Thanks

Fix checked in for py3k (r67612), release26-maint (r67613),
release30-maint (r67615) branches as well as trunk (r67614).

now 2008-12-06 trunk fail on linux

may be patch is not applied correctly :
http://svn.python.org/view/python/trunk/setup.py?rev=67614&view=auto

        libraries = gdbm_libs ) )
?   exts.append( Extension('dbm', ['dbmmodule.c'],
?     define_macros=[('HAVE_GDBM_NDBM_H',None)],
?     libraries = ['gdbm'] ) )
  elif db_incs is not None:

Roumen> may be patch is not applied correctly :
Roumen> http://svn.python.org/view/python/trunk/setup.py?rev=67614&view=auto

Roumen>         libraries = gdbm_libs ) )
Roumen> ?   exts.append( Extension('dbm', ['dbmmodule.c'],
Roumen> ?     define_macros=[('HAVE_GDBM_NDBM_H',None)],
Roumen> ?     libraries = ['gdbm'] ) )
Roumen>   elif db_incs is not None:

Yeah, muffed it. I should have a fix checked in shortly.

Skip

I'm not sure that recent commits in trunk are correct.

Please confirm that build of dbm module with "Berkeley DB" is deprecated.

Note that exist another module gdbm[gdbmmodule.c].

Roumen> I'm not sure that recent commits in trunk are correct.

Roumen> Please confirm that build of dbm module with "Berkeley DB" is
Roumen> deprecated.

Can you explain how you think they are incorrect? The code to decide how to
build the dbm module looks like this:

    if we have ndbm.h:
        build dbm using ndbm
    else if we have a gdbm library:
        if we have gdbm/ndbm.h:
            build dbm using gdbm
        else if we have gdbm-ndbm.h:
            build dbm using gdbm
    else if we found Berkeley includes:
        build dbm using Berkeley db.

This overall structure agrees with that in the Python 2.4 and 2.5 setup.py.
The only thing that changed (in theory) was that there's an extra check for
gdbm-ndbm.h after we've already decided to build dbm with gdbm. The
preference is still ndbm before gdbm before BerkDB. That's not to say it's
correct, but nothing's changed as far as I can tell.

Maybe it should look more like this:

    dbmext = None
    if we have ndbm.h:
        ...
        dbmext = Extension(...)

    if dbmext is None and we have a gdbm library:
        if we have gdbm/ndbm.h:
            ...
            dbmext = Extension(...)
        else if we have gdbm-ndbm.h:
            ...
            dbmext = Extension(...)

    if dbmext is None and we found Berkeley includes:
        ...
        dbmext = Extension(...)

    if dbmext is not None:
        exts.append(dbmext)

That still leaves the order ndbm before gdbm before BerkDB.

Skip

The old if statement was "flat":
if find_file("ndbm.h", inc_dirs, []) is not None:
<CODE>
elif (self.compiler.find_library_file(lib_dirs, 'gdbm')
and find_file("gdbm/ndbm.h", inc_dirs, []) is not None):
<CODE>
elif db_incs is not None:
<CODE>
else:
miss
If the second case fail, try third and if succeed build dbm with
"berkeley db".

The new if statement contain nested if:
Now the second case is only "self.compiler.find_library_file(lib_dirs,
'gdbm'):" and if succeed (my case) =>
Failed to find the necessary bits to build these modules:
... dbm ...
Note the build system lack headers "gdbm/ndbm.h", "gdbm-ndbm.h".

To restore previous I add copy of code from third case as new case in
nested if:
===============
'dbm', ['dbmmodule.c'],
define_macros=[('HAVE_GDBM_DASH_NDBM_H',None)],
libraries = gdbm_libs ) )
+ elif db_incs is not None:
+ exts.append( Extension('dbm', ['dbmmodule.c'],
+ library_dirs=dblib_dir,
+ runtime_library_dirs=dblib_dir,
+ include_dirs=db_incs,
+ define_macros=[('HAVE_BERKDB_H',None),
+ ('DB_DBM_HSEARCH',None)],
+ libraries=dblibs))
else:
missing.append('dbm')
elif db_incs is not None:
===============
Note that above is not proposed patch as I don't like nested ifs and
code duplicate.
Is possible the checks to be in a new separate function ?

Roumen> The old if statement was "flat":
...

Okay, I understand now.

Roumen> The new if statement contain nested if:
Roumen> Now the second case is only "self.compiler.find_library_file(lib_dirs,
Roumen> 'gdbm'):" and if succeed (my case) =>
Roumen> Failed to find the necessary bits to build these modules:
Roumen> ... dbm ...
Roumen> Note the build system lack headers "gdbm/ndbm.h", "gdbm-ndbm.h".

In my mind that simply means we haven't accounted for how your system has
organized its header files. It has a libgdbm so it probably has some sort
of gdbm header file unless you have failed to install some package.

I'm beginning to think this area needs more work. Let's leave this ticket
closed. I'll open a new one so we can figure out the best way to tackle
this. In the meantime can you figure out why your system has a gdbm library
but apparently no gdbm headers (or different ones than we are currently
testing for)?

Thx,

Skip

Skip> I'm beginning to think this area needs more work. Let's leave
Skip> this ticket closed. I'll open a new one so we can figure out the
Skip> best way to tackle this.

http://bugs.python.org/issue4587

Skip

This is still an issue in Python 2.7.3 but there is a quick manual workaround. I know it's trivial and one can easily develop it from what is said in the thread or maybe looking at the patches, but for reference this is a nice recipe as oppose to digging through many messages

build as usual, but redirect the output/error stream in a file, for example if your shell is bash (I find this to always be a good idea):

make > make.log 2>&1

Towards the end of make.log there will be the following message. If you don't see this message, you don't have this problem, but possibly a different one:

Failed to build these modules:
dbm

execute
grep "\-o .*/dbmmodule.o" make.log

This will find a compiler line. Cut, paste and and execute that command

grep "\-o .*/dbm.so" make.log

This will find another compiler line. Cut, paste and and execute that command, ADDING (this is essential) -lgdbm_compat

(optional) you may want to remove dbm_failed.so (in the same directory where the previous bullet creates dbm.so)

Navigation Menu

Repository navigation

Error to build _dbm module during make #48733

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Error to build _dbm module during make #48733

Description

Activity

legerf commented on Dec 1, 2008

smontanaro commented on Dec 2, 2008

smontanaro commented on Dec 2, 2008

smontanaro commented on Dec 2, 2008

legerf commented on Dec 2, 2008

legerf commented on Dec 2, 2008

smontanaro commented on Dec 2, 2008

legerf commented on Dec 2, 2008

smontanaro commented on Dec 2, 2008

legerf commented on Dec 2, 2008

legerf commented on Dec 4, 2008

6 remaining items

AndyP commented on Dec 5, 2008

legerf commented on Dec 5, 2008

flub commented on Dec 6, 2008

smontanaro commented on Dec 6, 2008

rpetrov commented on Dec 6, 2008

rpetrov commented on Dec 6, 2008

smontanaro commented on Dec 7, 2008

rpetrov commented on Dec 7, 2008

smontanaro commented on Dec 7, 2008

rpetrov commented on Dec 7, 2008

smontanaro commented on Dec 8, 2008

smontanaro commented on Dec 8, 2008

ddventoucaredu commented on Feb 8, 2013

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions