Skip to content

Add a file_digest() function in hashlib #89313

Closed
Closed
Add a file_digest() function in hashlib#89313
Assignees
tiran
Labels
3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@tarekziade
tarekziade
mannequin
opened on Sep 9, 2021
BPO 45150
Nosy @gpshead, @tiran, @tarekziade, @SonOfLilit, @miss-islington
PRs
  • bpo-45150: add a simple file_digest helper #28252
  • bpo-45150: draft implementation only for sha224,sha256 #31928
  • bpo-45150: Add hashlib.file_digest() for efficient file hashing #31930
  • bpo-45150: Fix testing under FIPS mode (GH-32046) #32046
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-09-09.10:03:17.355>
labels = ['type-feature', 'library', '3.11']
title = 'Add a file_digest() function in hashlib'
updated_at = <Date 2022-03-22.15:41:04.324>
user = 'https://github.com/tarekziade'

    bugs.python.org fields:

    activity = <Date 2022-03-22.15:41:04.324>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-09-09.10:03:17.355>
creator = 'tarek'
dependencies = []
files = []
hgrepos = []
issue_num = 45150
keywords = ['patch']
message_count = 14.0
messages = ['401457', '401461', '401462', '415138', '415139', '415141', '415320', '415321', '415324', '415326', '415336', '415356', '415754', '415793']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'tarek', 'python-dev', 'Aur.Saraf', 'miss-islington']
pr_nums = ['28252', '31928', '31930', '32046']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45150'
versions = ['Python 3.11']

    Activity

    1. tarekziade commented on Sep 9, 2021

      @tarekziade
      tarekziademannequin
      on Sep 9, 2021
      MannequinAuthor

      I am proposing the addition of a very simple helper to return the hash of a file.

    2. tarekziade
      self-assigned this
      on Sep 9, 2021
    3. tarekziade
      added
      stdlibStandard Library Python modules in the Lib/ directory
      on Sep 9, 2021
    4. tarekziade
      self-assigned this
      on Sep 9, 2021
    5. tarekziade
      added
      stdlibStandard Library Python modules in the Lib/ directory
      on Sep 9, 2021
    6. serhiy-storchaka
      added
      3.11only security fixes
      type-featureA feature request or enhancement
      on Sep 9, 2021

    7. tiran commented on Sep 9, 2021

      @tiran
      tiran
      on Sep 9, 2021
      Member

      Hey Tarek, long time no see!

      • the _sha256 module is optional, can be disabled and is not available in some distributions.

      • I also don't like to use sha256 has default. It's slow, even slower than sha512. Any default makes it also harder to upgrade to a better, more secure default in the future.

      • like hmac.new() a file_digest() should accept PEP-452-compatible arguments and hash name as digstmod argument, not just a callable.

      • a filename argument prevents users from passing in file-like objects like BytesIO.

      • 4096 bytes chunk size is very conservative. The call overhead for read() and update() may dominate the performance of the function.

      • The hex argument feels weird.

      In a perfect world, the hash and hmac objects should get an "update_file" method. The OpenSSL-based hashes could even release the GIL and utilize OpenSSL's BIO layer to avoid any Python overhead.

    8. tarekziade commented on Sep 9, 2021

      @tarekziade
      tarekziademannequin
      on Sep 9, 2021
      MannequinAuthor

      Hey Christian, I hope things are well for you!
      Thanks for all the precious feedback, I'll rework the patch accordingly

    9. SonOfLilit commented on Mar 14, 2022

      @SonOfLilit
      SonOfLilitmannequin
      on Mar 14, 2022
      Mannequin

      Tarek,

      Are you still working on this? Would you like me to take over?

      Aur

    10. tarekziade commented on Mar 14, 2022

      @tarekziade
      tarekziademannequin
      on Mar 14, 2022
      MannequinAuthor

      @aur, go for it, I started to implement it and got lost into the details for each backend..

    11. SonOfLilit commented on Mar 14, 2022

      @SonOfLilit
      SonOfLilitmannequin
      on Mar 14, 2022
      Mannequin

      OK, I'll give it a go.

    12. SonOfLilit commented on Mar 16, 2022

      @SonOfLilit
      SonOfLilitmannequin
      on Mar 16, 2022
      Mannequin

      PR contains a draft implementation, would appreciate some review before I implement the same interface on all builtin hashes as well as OpenSSL hashes.

    13. 5 remaining items

    14. tiran
      assigned
      tiran
      and unassigned
      tarekziade
      on Mar 16, 2022

    15. SonOfLilit commented on Mar 16, 2022

      @SonOfLilit
      SonOfLilitmannequin
      on Mar 16, 2022
      Mannequin

      I don't think HMAC of a file is a common enough use case to support, but I have absolutely no problem conceding this point, the cost of supporting it is very low.

      I/O in C is a world of pain in general. In the specific case of io.RawIOBase objects (non-buffered binary files) to my understanding it's not that terrible (am I right? Does my I/O code work as-is?). To my understanding, providing a fast path just for this case that calculates the hash without taking the GIL for every chunk would be very nice to have for many use cases.

      Now, we could just be happy with file_digest() having an if for isinstance(io.RawIOBase) that chooses a fast code path silently. But since non-buffered binary files are so hard to tell apart from other types of file-like objects, as a user of this code I would like to have a way to say "I want the fast path, please raise if I accidentally passed the wrong things and got the regular path". We could have file_digest('sha256', open(path, 'rb', buffered=0), ensure_fast_io=True), but I think for this use case raw_file_digest('sha256', open(path, 'rb', buffered=0)) is cleaner.

      In all other cases you just call file_digest(), probably get the Python I/O and not the C I/O, and are still happy to have that loop written for you by someone who knows what they're doing.

      For the same reason I think the fast path should only support hash names and not constructors/functions/etc', which would complicate it because new-object-can-be-accessed-without-GIL wouldn't necessarily apply.

      Does this make sense?

    16. miss-islington commented on Mar 22, 2022

      @miss-islington
      miss-islington
      on Mar 22, 2022
      Contributor

      New changeset 4f97d64 by Christian Heimes in branch 'main':
      bpo-45150: Add hashlib.file_digest() for efficient file hashing (GH-31930)
      4f97d64

    17. tiran commented on Mar 22, 2022

      @tiran
      tiran
      on Mar 22, 2022
      Member

      New changeset e03db6d by Christian Heimes in branch 'main':
      bpo-45150: Fix testing under FIPS mode (GH-32046)
      e03db6d

    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    19. pablogsal commented on Aug 13, 2022

      @pablogsal
      pablogsal
      on Aug 13, 2022
      Member

      @tiran Can you made a PR adding the file_digest to the 3.11 what's new, please?

    20. tiran
      added a commit that references this issue on Aug 13, 2022

      pythongh-89313: Add hashlib.file_digest to whatsnew 3.11

      603285c
    21. miss-islington
      added 2 commits that reference this issue on Aug 13, 2022

      gh-89313: Add hashlib.file_digest to whatsnew 3.11 (GH-95965)

      0b329f4

      pythongh-89313: Add hashlib.file_digest to whatsnew 3.11 (pythonGH-95965

      7db2c1a
    22. tiran
      closed this as completedon Aug 13, 2022
    23. miss-islington
      added a commit that references this issue on Aug 13, 2022

      gh-89313: Add hashlib.file_digest to whatsnew 3.11 (GH-95965)

      d89f5fe

    24. calestyo commented on Jun 23, 2023

      @calestyo
      calestyo
      on Jun 23, 2023
      Contributor

      Hey folks.

      I know this is closed and perhaps I should simply file a new request... but would you consider to extend the interface of that function to (efficiently) calculate a file's hashsum for multiple algorithms (i.e. without reading it once for every algo)?

      One could perhaps do so by accepting some array for digest and return a dict where the alogo name is the key and the hashvalue the value.

      Or perhaps something smarter ^^

      Cheers,
      Chris.

    25. gpshead commented on Jun 23, 2023

      @gpshead
      gpshead
      on Jun 23, 2023
      Member

      Please file a new feature request issue here for that.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions