pickle and pickletools cli interface doesn't close input and output file.

@avassalotti

BPO	41395
Nosy	@avassalotti, @serhiy-storchaka, @corona10, @tirkarthi, @amiremohamadi, @Foxyblue, @achhina
PRs	bpo-41395: Close file objects in pickle and pickletools #21676 bpo-41395: use context manager to close filetype objects #21702 gh-85567: Register a cleanup function to close files for FileType objects in argparse #32257

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-07-25.14:12:10.580>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.10', 'library']
title = "pickle and pickletools cli interface doesn't close input and output file."
updated_at = <Date 2022-04-05.13:05:47.595>
user = 'https://github.com/tirkarthi'

bugs.python.org fields:

activity = <Date 2022-04-05.13:05:47.595>
actor = 'achhina'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-07-25.14:12:10.580>
creator = 'xtreak'
dependencies = []
files = []
hgrepos = []
issue_num = 41395
keywords = ['patch', 'easy']
message_count = 2.0
messages = ['374269', '416776']
nosy_count = 8.0
nosy_names = ['alexandre.vassalotti', 'python-dev', 'serhiy.storchaka', 'corona10', 'xtreak', 'Amir', 's.williams-wynn', 'achhina']
pr_nums = ['21676', '21702', '32257']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41395'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

pickle and pickletools use argparse with FileType which is not automatically closed. Other cli interfaces like json [0], ast [1] use context manager to close filetype objects.

pickle :

cpython/Lib/pickle.py

Line 1799 in af08db7

'pickle_file', type=argparse.FileType('br'),

mypickle

>> import pickle
>> with open("mypickle", "wb") as f: pickle.dump({"a": 1}, f)

./python -Wall -m pickle mypickle
{'a': 1}
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='mypickle'>

pickletools :

cpython/Lib/pickletools.py

Lines 2850 to 2855 in af08db7

    
           parser.add_argument( 
        
               'pickle_file', type=argparse.FileType('br'), 
        
               nargs='*', help='the pickle file') 
        
           parser.add_argument( 
        
               '-o', '--output', default=sys.stdout, type=argparse.FileType('w'), 
        
               help='the file where the output should be written')

./python -Wall -m pickletools mypickle -o mypickle.py
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='mypickle'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='mypickle.py' mode='w' encoding='UTF-8'>

[0]

cpython/Lib/json/tool.py

Line 61 in af08db7

with options.infile as infile, options.outfile as outfile:

[1]

cpython/Lib/ast.py

Line 1510 in af08db7

with args.infile as infile:

Hi,

First-time contributor here, I've made a patch in follow-up to the discussions that happened in Amir's patch in regards to this. I'd appreciate it if someone would be able to take a look and review it!

#32257

Fix pickle and pickletools not closing files

@serhiy-storchaka

@serhiy-storchaka The issue seems to be resolved in this PR.

pythongh-85567: Fix resouce warnings in pickle and pickletools CLIs

The pickletools CLI is especially treacherous. It leaks the files if you specify the -t argument. It leaks the output file if you only specify the output file argument. And of course it leaks the files in case of argument errors, for example specifying non-integer indent level or non-writeable output file.

The only reliable way to handle this is to get rid of FileType and open and close files explicitly. PR #113618 does this.

gh-85567: Fix resouce warnings in pickle and pickletools CLIs (GH-113618

pythongh-85567: Fix resouce warnings in pickle and pickletools CLIs (p…

[3.11] gh-85567: Fix resouce warnings in pickle and pickletools CLIs (G…

[3.12] gh-85567: Fix resouce warnings in pickle and pickletools CLIs (G…

pythongh-85567: Fix resouce warnings in pickle and pickletools CLIs (p…

Navigation Menu

Repository navigation

pickle and pickletools cli interface doesn't close input and output file. #85567

Linked PRs

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	parser.add_argument(
	'pickle_file', type=argparse.FileType('br'),
	nargs='*', help='the pickle file')
	parser.add_argument(
	'-o', '--output', default=sys.stdout, type=argparse.FileType('w'),
	help='the file where the output should be written')

pickle and pickletools cli interface doesn't close input and output file. #85567

Description

Linked PRs

Activity

tirkarthi commented on Jul 25, 2020

achhina commented on Apr 5, 2022

9 remaining items

furkanonder commented on Nov 18, 2022

serhiy-storchaka commented on Jan 1, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions