Tarfile using fstat on GZip file object #66658

Activity

1. bartolsthoorn commented on Sep 23, 2014

   

   bartolsthoornmannequin

   on Sep 23, 2014

   MannequinAuthor

   More actions

   CPython tarfile gettarinfo method uses fstat to determine the size of a file (using its fileobject). When that file object is actually created with Gzip.open (so a GZipfile), it will get the compressed size of the file. The addfile method will then continue to read the uncompressed data of the gzipped file, but will read too few bytes, resulting in a tar of incomplete files.

   I suggest checking the file object class before using fstat to determine the size, and raise a warning if it's a gzip file.

   To clarify, this only happens when adding a GZip file object to tar. I know that it's not a really common scenario, and the problem is really that GZip file size can only properly be determined by uncompressing and reading it entirely, but I think it's nice to not fail without warning.

   So this is an example that is failing:

   import tarfile
   c = io.BytesIO()
   with tarfile.open(mode='w', fileobj=c) as tar:
     for textfile in ['1.txt.gz', '2.txt.gz']:
       with gzip.open(textfile) as f:
         tarinfo = tar.gettarinfo(fileobj=f)
         tar.addfile(tarinfo=tarinfo, fileobj=f)
     data = c.getvalue()
   return data
   

   Instead this reads the proper filesize and writes the files to a tar:

   import tarfile
   c = io.BytesIO()
   with tarfile.open(mode='w', fileobj=c) as tar:
     for textfile in ['1.txt.gz', '2.txt.gz']:
       with gzip.open(textfile) as f:
         buff = f.read()
         tarinfo = tarfile.TarInfo(name=f.name)
         tarinfo.size = len(buff)
         tar.addfile(tarinfo=tarinfo, fileobj=io.BytesIO(buff))
     data = c.getvalue()
   return data
   
2. bartolsthoorn

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 23, 2014
3. BreamoreBoy commented on Mar 23, 2015

   

   BreamoreBoymannequin

   on Mar 23, 2015

   Mannequin

   More actions

   msg227328 states "it's not a really common scenario" but I believe we must still allow for it, what do others think?
4. vadmium commented on Mar 23, 2015

   

   vadmium

   on Mar 23, 2015

   Member

   More actions

   I think a warning in the documentation might be helpful.

   However a special check in the code doesn’t seem right. Would you check for LZMAFile and BZ2File as well? Some of the other attributes (modification time, owner, etc) may be useful even for a GzipFile, and the programmer can just overwrite the file size attribute if necessary.
5. vadmium commented on Apr 20, 2015

   

   vadmium

   on Apr 20, 2015

   Member

   More actions

   I am posting a documentation patch which I hope should clarify that objects like GzipFile won’t work automatically with gettarinfo(). It also has other modifications to address bpo-21996 (name must be text) and help with bpo-22208 (clarify non-OS files won’t work).
6. vadmium

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   on Apr 20, 2015
7. vadmium

   assigned

   docspython

   on Apr 20, 2015
8. python-dev commented on Feb 20, 2016

   

   python-devmannequin

   on Feb 20, 2016

   Mannequin

   More actions

   New changeset 94a94deaf06a by Martin Panter in branch '3.5':
   Issues bpo-22468, bpo-21996, bpo-22208: Clarify gettarinfo() and TarInfo usage
   https://hg.python.org/cpython/rev/94a94deaf06a

   New changeset e66c476b25ec by Martin Panter in branch 'default':
   Issue bpo-22468: Merge gettarinfo() doc from 3.5
   https://hg.python.org/cpython/rev/e66c476b25ec

   New changeset 9d5217aaea13 by Martin Panter in branch '2.7':
   Issues bpo-22468, bpo-21996, bpo-22208: Clarify gettarinfo() and TarInfo usage
   https://hg.python.org/cpython/rev/9d5217aaea13
9. vadmium commented on Feb 20, 2016

   

   vadmium

   on Feb 20, 2016

   Member

   More actions

   Hoping my clarification in the documentation is enough to call this fixed
10. vadmium

    closed this as completedon Feb 20, 2016
11. ezio-melotti

    transferred this issue fromon Apr 10, 2022