Repository navigation
Allow incremental I/O to blobs in sqlite3 #69093
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2015-08-21.08:05:21.911>
labels = ['extension-modules', 'type-feature', '3.8']
title = 'Allow incremental I/O to blobs in sqlite3'
updated_at = <Date 2022-01-20.12:42:10.486>
user = 'https://bugs.python.org/jimminter'bugs.python.org fields:
activity = <Date 2022-01-20.12:42:10.486>
actor = 'erlendaasland'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2015-08-21.08:05:21.911>
creator = 'jim_minter'
dependencies = []
files = ['41798', '44142', '46531', '46549']
hgrepos = []
issue_num = 24905
keywords = ['patch']
message_count = 28.0
messages = ['248945', '259262', '259520', '272873', '272883', '273052', '286957', '287040', '287065', '287081', '287162', '288515', '288517', '288519', '288724', '288725', '297205', '315450', '329457', '372225', '399813', '409494', '409580', '409584', '409591', '409592', '410920', '411021']
nosy_count = 9.0
nosy_names = ['ghaering', 'berker.peksag', 'serhiy.storchaka', 'jim_minter', 'waldhol', 'palaviv', 'eamanu', 'erlendaasland', 'rmast']
pr_nums = ['271', '30356', '30680']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24905'
versions = ['Python 3.8']
Activity
jimminter commentedon Aug 21, 2015
SQLite supports incremental I/O to blobs, i.e. the capability to stream reads and writes to blobs without having to load the entire blob into memory first. See https://www.sqlite.org/c3ref/blob_open.html for more details on the C API.
It'd be nice if it were possible to do this in Python using sqlite3 (it is already possible with apsw).
palaviv commentedon Jan 30, 2016
I opened a pull request for blob support in the pysqlite github repository:
ghaering/pysqlite#93
I will do the needed changes for python3 and will post a patch soon.
palaviv commentedon Feb 3, 2016
I did the needed changes for the pull request at pysqlite for porting to python3. I will continue updating the patch if there will be changes in pysqlite and vice versa.
palaviv commentedon Aug 16, 2016
Pinging as mentioned in the devguide.
serhiy-storchaka commentedon Aug 16, 2016
Added comments on Rietveld. The patch contains typos and violates PEP-7 and PEP-12. And there are questions about API.
palaviv commentedon Aug 18, 2016
Thanks for the review Serhiy. Attached is the updated patch after the changes.
palaviv commentedon Feb 4, 2017
Pinging again. I think this would be a great enhancement to the sqlite module.
serhiy-storchaka commentedon Feb 5, 2017
Sorry Aviv, I just forget about this issue.
Added new comments on Rietveld. Many lines in sqlite3.rst still are too long.
It would be worth to ask other developers about wanted interface.
palaviv commentedon Feb 5, 2017
Thanks for the CR Serhiy. Attached is a new patch after the fixes from the CR.
What other developers should I ask? The interface is file like and is the same as apsw.
serhiy-storchaka commentedon Feb 6, 2017
apsw has different order of arguments for blobopen(), all arguments are mandatory, and the parameter for read/write mode is called "writeable" rather than "readonly". This part of API needs to be discussed. Ask for discussion on mailing lists: Python-Ideas and maybe Python-List.
palaviv commentedon Feb 6, 2017
Uploading patch after fixes from berker CR.
The
blob_openAPI can can have the following options:a. No default (mandatory parameter).
b. default read-only
c. default write-only
I don't think that there is enough differences between the possible API's to justify sending a message to the mailing lists.
palaviv commentedon Feb 24, 2017
I opened a PR in github. I tagged some other developers if that will not start a discussion on the API I will post in the python-ideas.
33 remaining items
gh-69093: Don't allow instantiation of sqlite3.Blob objects (GH-91570)
gh-69093: improve sqlite3.Connection.blobopen() error handling (GH-91571
gh-69093: Add context manager support to sqlite3.Blob (GH-91562)
gh-69093: sqlite3 blob doc amendments (GH-91561)
gh-69093: Add indexing and slicing support to sqlite3.Blob (#91599)
erlend-aasland commentedon Apr 22, 2022
Incremental I/O for blobs is now supported in the sqlite3 extension module in the upcoming Python 3.11 release. Thanks a lot to @palaviv, @JelleZijlstra, and everyone else who contributed by commenting and reviewing.
IMO, the implementation can benefit from further tweaks, but let's do that in separate issues if needed.
erlend-aasland commentedon Apr 22, 2022
cc. @eamanu, @nightlark
pythongh-69093: Fix Setup.local.in rule for _sqlite3
gh-69093: Fix Setup.local.in rule for _sqlite3 (GH-93380)
pythongh-69093: Fix Setup.local.in rule for _sqlite3 (pythonGH-93380)
gh-69093: Fix Setup.local.in rule for _sqlite3 (GH-93380)