Skip to content

Add the iterator protocol to dbm modules #49986

Closed
Closed
Add the iterator protocol to dbm modules#49986
Labels
extension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

Description

@akitada
akitada
mannequin
opened on Apr 11, 2009
BPO 5736
Nosy @loewis, @rhettinger, @merwok, @akitada
Superseder
  • bpo-9523: Improve dbm modules
    		•
    Files
  • issue5736.diff: iter(dbm.keys())
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2011-02-12.20:17:43.215>
created_at = <Date 2009-04-11.13:47:52.382>
labels = ['extension-modules', 'type-feature']
title = 'Add the iterator protocol to dbm modules'
updated_at = <Date 2011-02-12.20:17:43.213>
user = 'https://github.com/akitada'

    bugs.python.org fields:

    activity = <Date 2011-02-12.20:17:43.213>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2011-02-12.20:17:43.215>
closer = 'eric.araujo'
components = ['Extension Modules']
creation = <Date 2009-04-11.13:47:52.382>
creator = 'akitada'
dependencies = []
files = ['19250']
hgrepos = []
issue_num = 5736
keywords = ['patch']
message_count = 14.0
messages = ['85856', '85859', '85867', '85878', '85888', '85889', '85928', '85931', '85944', '85946', '91339', '118874', '123358', '128465']
nosy_count = 6.0
nosy_names = ['loewis', 'rhettinger', 'eric.araujo', 'akitada', 'foobaron', 'ysj.ray']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '9523'
type = 'enhancement'
url = 'https://bugs.python.org/issue5736'
versions = []

    Activity

    1. akitada commented on Apr 11, 2009

      @akitada
      akitadamannequin
      on Apr 11, 2009
      MannequinAuthor

      In Python 2.6, dbm modules othar than bsddb don't support the iterator
      protocol.

      >>> import dbm
>>> d = dbm.open('spam.dbm', 'c')
>>> for k in range(5): d["key%d" % k] = "value%d" % k
... 
>>> for k in d: print k, d[k]
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'dbm.dbm' object is not iterable

      Adding iterator support would make dbm modules more convenient and
      easier to use.

    2. akitada
      added
      extension-modulesC modules in the Modules dir
      type-featureA feature request or enhancement
      on Apr 11, 2009

    3. akitada commented on Apr 11, 2009

      @akitada
      akitadamannequin
      on Apr 11, 2009
      MannequinAuthor

      Attached is a patch that adds the iterator protocol.
      Now it can be interated through like:

      >>> for k in d: print k, d[k]
... 
key1 vale1
key3 vale3
key0 vale0
key2 vale2
key4 vale4

      The problem is there is no way to get the internal pointer back to the
      start. So Once it reached to the end, you are done.

      >>> for k in d: print k, d[k]
...

      The solution to this would be:

      • Add a method to get the pointer back to the start
        (with {first,next}key API)
      • Add a method that returns a generator

    4. akitada commented on Apr 11, 2009

      @akitada
      akitadamannequin
      on Apr 11, 2009
      MannequinAuthor

      Revised patch adds firstkey and nextkey to dbm.
      Now the internal pointer can be reset with firstkey.

    5. loewis commented on Apr 11, 2009

      @loewis
      loewismannequin
      on Apr 11, 2009
      Mannequin

      Would you like to fix gdbm as well?

    6. akitada commented on Apr 12, 2009

      @akitada
      akitadamannequin
      on Apr 12, 2009
      MannequinAuthor

      Here's another patch which addsd iter to dbm and gdbm.

      Note that dbm and gdbm C API is a little different.
      gdbm_nextkey requires key for its argument, dbm_nextkey don't.
      So I had to use for gdbm an static variable that points to the current
      position.
      Now iterator in gdbm and dbm works differently.

      >>> import dbm
>>> d = dbm.open('foo', 'n')
>>> d['k1'] = 'v1';d['k2'] = 'v2';
>>> for i in d: print i; break
... 
k1
>>> for i in d: print i
... 
k2
>>> for i in d: print i
... 


>>> import gdbm
>>> gd = gdbm.open('foo.gdbm', 'n')
>>> gd['k1'] = 'v1';gd['k2'] = 'v2';
>>> for i in gd: print i; break
... 
k2
>>> for i in gd: print i
for i in gd: print i
... 
k1
>>> for i in gd: print i
... 
k2
k1

    7. akitada commented on Apr 12, 2009

      @akitada
      akitadamannequin
      on Apr 12, 2009
      MannequinAuthor

      Of course iter should work in the same way in all dbm modules.
      iter in dbm/gdbm should work like dumbdbm's iter.

      >>> dumb = dumbdbm.open('foo', 'n')
>>> dumb['k1'] = 'v1';dumb['k2'] = 'v2';
>>> for i in dumb: print i; break
... 
k2
>>> for i in dumb: print i
for i in dumb: print i
... 
k2
k1
>>> for i in dumb: print i
for i in dumb: print i
... 
k2
k1

    8. smontanaro commented on Apr 12, 2009

      @smontanaro
      smontanaro
      on Apr 12, 2009
      Contributor

      Akira> Note that dbm and gdbm C API is a little different. gdbm_nextkey
      Akira> requires key for its argument, dbm_nextkey don't. So I had to
      Akira> use for gdbm an static variable that points to the current
      Akira> position.

      I don't think this is going to fly. A static variable is not thread-safe.
      What's worse, even in a non-threaded environment you might want to iterate
      over the gdbm file simultaneously from two different places.

    9. smontanaro commented on Apr 13, 2009

      @smontanaro
      smontanaro
      on Apr 13, 2009
      Contributor

      skip> What's worse, even in a non-threaded environment you might want to
      skip> iterate over the gdbm file simultaneously from two different
      skip> places.

      Or iterate over two different gdbm files simultaneously.

    10. loewis commented on Apr 13, 2009

      @loewis
      loewismannequin
      on Apr 13, 2009
      Mannequin

      I agree with Skip that using a static variable is not appropriate. The
      proper solution probably would be to define a separate gdbm_iter object
      which always preserves the last key returned.

    11. akitada commented on Apr 13, 2009

      @akitada
      akitadamannequin
      on Apr 13, 2009
      MannequinAuthor

      Yes, using a static variable there is wrong and
      actually I'm now working on "dbm_iterobject" just as Martin suggested.

      dbm iterator should behave just like one in dict.
      I think I can use Objects/dictobject.c as a good example for this.

      Attached is minimal tests for dbm iterator.

    12. foobaron commented on Aug 6, 2009

      @foobaron
      foobaronmannequin
      on Aug 6, 2009
      Mannequin

      Another reason this issue is really important, is that the lack of a
      consistent iter() interface for dbm.* makes shelve iteration not
      scalable; i.e. trying to iterate on a Shelf will run self.dict.keys() to
      load the entire index into memory. This seems contrary to a primary
      purpose of shelve, namely to store the index on-disk so as to avoid
      having to keep the whole index in memory.

      I suspect that for most users, shelve is the main way they will access
      the dbm.* interfaces. Therefore, fixing the dbm.* interfaces so that
      shelve is scalable seems like an important need.

      Once dbm and gdbm support the iterator protocol, it will be trivial to
      add an __iter__() method to shelve.Shelf, that simply returns
      iter(self.dict).

    13. akitada commented on Oct 16, 2010

      @akitada
      akitadamannequin
      on Oct 16, 2010
      MannequinAuthor

      This patch just uses PyObject_GetIter to get an iter.
      (I just copied the idea from bpo-9523)

    14. merwok commented on Dec 4, 2010

      @merwok
      merwok
      on Dec 4, 2010
      Member

      This may be superseded by bpo-9523. There are comments and patches in both issues, so I’m not closing either as duplicate of the other.

    15. merwok commented on Feb 12, 2011

      @merwok
      merwok
      on Feb 12, 2011
      Member

      bpo-9523 has a more comprehensive patch in progress, adding __iter__ and other mapping methods, so I’m closing this one.

    16. merwok
      closed this as completedon Feb 12, 2011
    17. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      extension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions