argparse.FileType opens a file and never closes it

@merwok

BPO	13824
Nosy	@merwok, @mitar, @matrixise, @MojoVampire, @remilapeyre, @septatrix
Files	patch_3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-01-19.12:25:39.321>
labels = ['type-bug', 'library']
title = 'argparse.FileType opens a file and never closes it'
updated_at = <Date 2021-07-25.09:27:11.812>
user = 'https://bugs.python.org/DavidLayton'

bugs.python.org fields:

activity = <Date 2021-07-25.09:27:11.812>
actor = 'Nils Kattenbeck'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-01-19.12:25:39.321>
creator = 'David.Layton'
dependencies = []
files = ['31852']
hgrepos = []
issue_num = 13824
keywords = ['patch']
message_count = 15.0
messages = ['151615', '152518', '152519', '152530', '166068', '198336', '198376', '215298', '341670', '341761', '341764', '341852', '342153', '342503', '398184']
nosy_count = 13.0
nosy_names = ['bethard', 'eric.araujo', 'mitar', 'Paolo.Elvati', 'manveru', 'Stefan.Pfeiffer', 'paul.j3', 'David.Layton', 'matrixise', 'josh.r', 'remi.lapeyre', 'sebix', 'Nils Kattenbeck']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13824'
versions = ['Python 2.7']

Linked PRs

argparse.FileType.__call__ opens the specified file and returns it. This is well documented as an anit-idiom in http://docs.python.org/howto/doanddont.html#exceptions.

"...a serious problem — due to implementation details in CPython, the file would not be closed when an exception is raised until the exception handler finishes; and, worse, in other implementations (e.g., Jython) it might not be closed at all regardless of whether or not an exception is raised."

Disregarding the above, handling a file which may or may not have been opened depending the users input requires a bit of boilerplate code compared to the usual with-open idiom.

Additionally, there is no way to prevent FileType from clobbering an existing file when used with write mode.

Given these issues and others, it seems to me that the usefulness of FileType is outweighed by propensity to encourage bad coding. Perhaps, it would be best if FileType (or some replacement) simply checked that the file exists (when such a check is appropriate), it can be opened in the specified mode, and, curry the call to open (i.e. return lambda: open(string, self._mode, self._bufsize))

Additionally, there is no way to prevent FileType from clobbering an existing file when used with write mode.

I think this deserves its own feature request: now that Python 3.3 has an “exclusive create” mode, argparse.FileType could gain support for it. Would you open that request?

Given these issues and others,

We have one issue and one missing feature; what are the other issues?

it seems to me that the usefulness of FileType is outweighed by propensity to encourage bad coding.

I think the problem is not as bad as you paint it. A great number of unclosed file handles may cause problem to some OSes, but that’s it. On the plus side, the fact that argparse accepts for its type argument any callable that can check and convert a string input is simple, clean and works. FileType is needed.

In packaging/distutils2 for example we have similar functions that return an open file object and never close it: the responsibility is at a higher level. Other packaging code calling these functions does so in a with statement. It is not evil by nature.

The problem here is that FileType may return stdin or stdout, so we can’t just always close the file object (or maybe we can, say using an atexit handler?).

Perhaps, it would be best if FileType (or some replacement) simply checked that the file exists

But then you’d run into race conditions. The only sure was to say if a file can be opened is to open it.

s/sure was/sure way/

Eric,

But then you’d run into race conditions. The only sure was to say if a file can be opened is to open it.

I think you misunderstand me. I am NOT suggesting that you open and close the file. I am saying that you should not open it in the first place.

If I cannot open the file at the point in my program where I actually want to open it, then fine, I can decide, in my own code, what to do.

I think the problem is not as bad as you paint it. A great number of unclosed file handles may cause problem to some OSes, but that’s it. On the plus side, the fact that argparse accepts for its type argument any callable that can check and convert a string input is simple, clean and works. FileType is needed.

Causing a problem on some OSes and not others is worse than causing a problem on all OSes as it increases the likelihood of buggy code passing tests and moving to production.

I think argparse is wonderful; I just think that by having FileType not open the file, the number of it's use cases is increased. As it stands now, I would prefer the just pass the argument as a string argument and handling the opening myself unless:

I wanted my program to open the file at the very beginning of the program (where one traditionally handles arg parsing)
I wanted to exit on the first, and only, attempt to open the file
I really did not care if the file closed properly--which, granted, is often the case with tiny scripts

The moment any of these is not true due to a change in requirements, I will have to refactor my code to use a filename arg. Where as if I start out with a bog-standard filename and open it myself, I can easily add the behaviour I want. I just don't see FileType as a big convenience. However, I do see that changing this would break backwards compatibility and would not want to see that happen. Perhaps a new FileNameType that does some basic, perhaps, optional checks would have wider use-cases.

I hope this helps.

David Layton

So I generally agree that FileType is not what you want for anything but quick scripts that can afford to either leave a file open or to close stdin and/or stdout. However, quick scripts are an important use case for argparse, so I don't think we should get rid of FileType.

What should definitely happen:

Someone should add some documentation explaining the risks of using FileType (e.g. forgetting to close the file, or closing stdin/stdout when you didn't mean to).

What could potentially happen if someone really wanted it:

Someone could create a "safe" replacement, e.g. a FileOpenerType that returns an object with open() and close() methods that do the right things (or whatever API makes sense).

In this patch I implement a FileContext class. It differs from FileType
in 2 key areas:

it returns a 'partial(open, filename, ...)'
it wraps '-' (stdin/out) in a dummy context protecting the file from closure.

The resulting argument is meant to be used as:

    with args.input() as f:
        f.read()
        etc

The file is not opened until value is called, and it will be closed after the block. stdin/out can also be used in this context, but without closing.

The signature for this class is the same as for FileType, with one added parameter, 'style'. (alternative name suggestions welcomed).

class argparse.FileContext(mode='r', bufsize=-1, encoding=None, errors=None, style='delayed')

The default behavior, "style='delayed'", is as described above.

"style='evaluate'" immediately calls the partial, returning an opened file. This is essentially the same as FileType, except for the stdin/out context wrapping.

"style='osaccess'", adds os.acccess testing to determine whether the 'delayed' file can be read or written. It attempts to catch the same sort of OS errors that FileType would, but without actually opening or creating the file.

Most of the added tests in test_argparse.py copy the FileType tests. I had to make some modifications to the testing framework to handle the
added levels of indirection.

I have not written the documentation changes yet.

A sample use case is:

    import argparse, sys
    p = argparse.ArgumentParser()
    p.add_argument('-d','--delayed', type=argparse.FileContext('r'))
    p.add_argument('-e','--evaluated', type=argparse.FileContext('r', style='evaluate'))
    p.add_argument('-t','--test', dest='delayed', type=argparse.FileContext('r', style='osaccess'))
    p.add_argument('-o','--output', type=argparse.FileContext('w', style='osaccess'), default='-')
    p.add_argument('--unused', type=argparse.FileContext('w', style='osaccess'),help='unused write file')
    args = p.parse_args()

    with args.output() as o:
        if args.delayed:
            with args.delayed() as f:
                print(f.read(), file=o)
        if args.evaluated:
            with args.evaluated as f:
                print(f.read(), file=o)
    # f and o will be closed if regular files
    # but not if stdin/out
    # the file referenced by args.unused will not be created

An alternative way to delay the file opening, is to return an instance that has a filename attribute, and has an open method. This can be compactly added to the FileContext that I defined in the previous patch. The FileContext provides the _ostest function (testing using os.access), and wrapper for stdin/out.

    class FileOpener(argparse.FileContext):
        # delayed FileType; alt to use of partial()
        # sample use:
        # with args.input.open() as f: f.read()
        def __call__(self, string):
            string = self._ostest(string)
            self.filename = string
            return self
        def open(self):
            return self.__delay_call__(self.filename)()
        file =  property(open, None, None, 'open file property')

From http://bugs.python.org/issue14156
"argparse.FileType for '-' doesn't work for a mode of 'rb'"

I learned that 'stdin/out' can be embedded in an 'open' by using the 'fileno()' and 'closefd=False' (so it isn't closed at the end of open).

With this, the dummy file context that I implemented in the previous patch, could be replaced with:

partial(open, sys.stdin.fileno(), mode=self._mode,..., closefd=False)

However, as Steven Bethard wrote in the earlier issue, setting up tests when stdin/out will be redirected is not a trivial problem. So I don't yet have a testable patch.

---------------
And just for the record, the 'osaccess' testing that I wrote earlier, probably should also test that proposed output file string is not already a directory.

Why not make FileType instance also a context manager, so you could do something like:

with arguments.input as f:
   assert arguments.input is f

For backwards compatibility, FileType would open eagerly as now, but it would be closed at exit from context manager.

We should just make sure we do not close stdout/stderr.

Mitar: argparse.FileType's __call__ (which is what "parses" the argument) returns whatever open (aka io.open) returns. Basically, post-parsing, there is nothing about the result of FileType that distinguishes it from a manually opened file (or when '-' is passed, from stdin/stdout). So it's already possible to do what you describe, simply by doing:

parser = argparse.ArgumentParser()
parser.add_argument('input', type=argparse.FileType())
arguments = parser.parse_args()

with arguments.input as f:
   assert arguments.input is f

That already "open[s] eagerly as now, but it would be closed at exit from context manager."

The original report did not like that you couldn't prevent clobbering of an existing file in write mode (no longer true, since you can pass a mode of 'x' just like you can with open), and did not like that the result of parsing was implicitly opened immediately without being controllable *immediately* via a context manager.

The only real solutions to that problem are either:

Making FileType (or a replacement) that is lazier in some way.
Making the argument namespace itself support context management

The proposal to check validity and return a curried call to open is problematic given the TOCTOU issues, but a curried version that performs little or no checks up front, but performs argparse style clean exits if the deferred open fails would be reasonable.

The alternative would be to integrate more heavily with the argument namespace, such that you could write code like:

with parser.parse_args() as arguments:

and in that scenario, the files would be opened immediately and stored on the namespace, but either only FileType, or any type result supporting context management, would be bulk __enter__-ed and bulk __exit__-ed upon exiting the with block.

As is, a user could do most of this with contextlib.ExitStack, but it's more to reimplement (and tricky to get right, and would still rely on argparse to clean up files 1 through n-1 if opening file n fails for whatever reason before parsing completes).

For information, bpo-33927 is related. I removed one of the FileType from json.tool argument parser to work around this behavior (https://github.com/python/cpython/pull/7865/files#diff-e94945dd18482591faf1e294b029a6afR44).

If paul.j3 does not have his patch anymore I volunteer to implement option 1 to have a better alternative in the stdlib.

So it's already possible to do what you describe, simply by doing:

I think there is an edge case here if a stdin/stdout is opened? It would get closed at exist from the context, no?

So I suggest that a slight extension of what open otherwise returns is returned which make sure context manager applies only to non-stdin/stdout handles.

pythongh-58032: Deprecate the argparse.FileType type converter (pytho…

pythongh-58032: Docs: Sort argparse alphabetically (pythonGH-125871)

Remove use of argparse.FileType

Do not use argparse.FileType

The docs don't really have an example for how to migrate away from this and keep - support. Having an example with a common mode or two would be helpful, I think. pylint detects usage of this now (code W4904), by the way, which is helpful.

I'm giving a talk on argparse at PyCon US next week (last minute addition from the waitlist, https://us.pycon.org/2026/schedule/presentation/165/, still hasn't appeared on the master schedule on the website as of 5/9), and I just now learned of the deprecation while reading the docs to prepare the talk. I'm trying now to think of a good alternative to advise people to use for the use case where a CLI program wants to read either one or more files or default to stdin.

The use cases to replace are the 0 or 1 files version

add_argument("infile", type=FileType("r"), default=sys.stdin, nargs="?")

and the similar 0 or many files version

add_argument("infiles", type=FileType("r"), default=[sys.stdin], nargs="*")

The first thing I can think of is to have a helper function that you use to open the file, or pass through the existing stdin/out/err file object. Something like

def open_file_or_io(file_path_or_io, **kwargs):
    if isinstance(file_path_or_io, IOBase):
        return nullcontext(file_path_or_io)
    return open(file_path_or_io, **kwargs)

Maybe also need to pair that with a couple helpers to use with the type argument of add_argument that will handle translating "-" to sys.stdin or out. Something like

def path_or_stream(stream):
    def _path_or_stream(arg):
        if arg == "-":
            return stream
        elif isinstance(arg, IOBase):
            return arg
        return Path(arg)
    return _path_or_stream

So you'd use it like

parser.add_argument("infile", type=path_or_stream(sys.stdin), default=sys.stdin)
...
with open_file_or_io(args.infile) as fobj:
    ....

Any reactions to that? I hesitate to stand up in front of a crowd and promote something I just made up and haven't used yet. But we need something to suggest to people now that the old way is officially deprecated.

@drcraig

@drcraig: I'm inclined to ask that the old way be undeprecated. I actually wrote my own subclass of argparse.FileType at work that works with contextlib.ExitStack, and it's extremely usable, aside from the new PendingDeprecationWarning, and integrating the feature into FileType itself is easy now that I've already done it once and explored the edge cases. Usage would boil down to:

with contextlib.ExitStack() as stack:
     # define rest of parser = argparse.ArgumentParser()
     parser.add_argument('infile', type=argparse.FileType(stack=stack)
     parser.add_argument('outfile', type=argparse.FileType("w", stack=stack)
     args = parser.parse_args()  # If any argument fails to parse here, all files opened by that point will be automatically closed

     # Either do everything you want to do with the files within the with, or
     # call stack.pop_all() before finishing the with to pass the stack elsewhere
     # for management

The behavior varies slightly for receiving - instead of a filename, only insofar as, instead of FileType.__call__ invoking self._stack.enter_context on the result of an open call, it instead invokes self._stack.callback(lambda: fileobj.closed or fileobj.flush()) when opened for write (and does nothing at all when it's opened for read), so sys.stdout (or its buffer, based on the mode) is guaranteed to be flushed (if not already closed) when the with block is exited.

The two main objections I can see are:

This is too FileType specific, and argparse could be made a context manager itself and implicitly manage all context-managers it produces. I'd say that this is a case where explicit is better than implicit (the developer has the option not to manage context manager types, and explicitly managing makes things less magical), and it makes FileType a good model for how to make your own context managed type converters for use with argparse
It's a recipe that integrates two very different modules (contextlib and argparse) as the recommended way of using argparse. But contextlib.ExitStack is an amazing tool that does its one job very well, and reinventing it piecemeal seems wrong, and beyond that, we're not averse to providing recipes combining modules elsewhere in the docs (the itertools recipes at this point are built on top of itertools, collections, contextlib, functools, heapq, math, and operator at this point).

The one thing it doesn't handle is cleaning up when a file is opened for write, and then a subsequent argument parsing failure closes it while leaving the empty file on disk. I'm largely okay with that; the user does know what they typed in after all, so the mystery file appearing isn't confusing.

Opened gh-150125 to propose restoring FileType.

fix: Cease usage of argparse.FileType

fix: Cease usage of argparse.FileType (#170)

Navigation Menu

Repository navigation

argparse.FileType opens a file and never closes it #58032

Linked PRs

32 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

argparse.FileType opens a file and never closes it #58032

Description

Linked PRs

Activity

DavidLayton commented on Jan 19, 2012

merwok commented on Feb 3, 2012

merwok commented on Feb 3, 2012

DavidLayton commented on Feb 3, 2012

bethard commented on Jul 21, 2012

paulj3 commented on Sep 23, 2013

paulj3 commented on Sep 25, 2013

paulj3 commented on Apr 1, 2014

mitar commented on May 7, 2019

MojoVampire commented on May 7, 2019

remilapeyre commented on May 7, 2019

mitar commented on May 8, 2019

32 remaining items

henryiii commented on Feb 6, 2026

drcraig commented on May 10, 2026

MojoVampire commented on May 16, 2026

MojoVampire commented on May 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions