explore hashlib use of the Linux Kernel CryptoAPI #91258

Activity

1. gpshead commented on Mar 23, 2022

   

   gpshead

   on Mar 23, 2022

   MemberAuthor

   More actions

   Linux kernels provide a CryptoAPI. This is a common place for platform specific hardware accelerated hash algorithms to be exposed to the user (especially on SoCs which often have non-standard hardware).

   https://www.kernel.org/doc/html/v4.10/crypto/userspace-if.html
   https://www.kernel.org/doc/html/v5.17/crypto/userspace-if.html
   https://www.chronox.de/libkcapi.html

   hashlib currently uses OpenSSL when possible for performance. We could also look at querying the kernel API. How to decide between the two implementations when both are present is something TBD.

   This would probably be best done via a configure time check for libkcapi?
2. gpshead

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Mar 23, 2022
3. tiran commented on Mar 23, 2022

   

   tiran

   on Mar 23, 2022

   Member

   More actions

   We don't need libkcapi. I added AF_ALG support a while ago:

   import binascii
   import os
   import socket
   
   with socket.socket(socket.AF_ALG, socket.SOCK_SEQPACKET, 0) as cfgsock:
       cfgsock.bind(("hash", "sha256"))
       opsock, _ = cfgsock.accept()
       with opsock:
           with open("/etc/os-release") as f:
               st = os.fstat(f.fileno())
               # blindly assumes that sendfile() exhausts the fd.
               os.sendfile(
                   opsock.fileno(), f.fileno(), offset=0, count=st.st_size
               )
               res = opsock.recv(512)
           print(binascii.hexlify(res))
4. gpshead commented on Mar 23, 2022

   

   gpshead

   on Mar 23, 2022

   MemberAuthor

   More actions

   Neat. I've never used the API, just filing a breadcrumb suggesting we see if it makes sense. Being I/O based, that even takes care of GIL releasing from Python. :)
5. tiran commented on Mar 23, 2022

   

   tiran

   on Mar 23, 2022

   Member

   More actions

   test_socket has examples for HMAC, AES-CBC, and AES-GCM.

   with self.create_alg('hash', 'hmac(sha1)') as algo:
       algo.setsockopt(socket.SOL_ALG, socket.ALG_SET_KEY, b"Jefe")
       op, _ = algo.accept()
       with op:
           op.sendall(b"what do ya want for nothing?")
           self.assertEqual(op.recv(512), expected)
6. tiran commented on Mar 23, 2022

   

   tiran

   on Mar 23, 2022

   Member

   More actions

   And sendfile() is zero-copy. Data does not have to leave Kernel space.
7. tiran commented on Mar 29, 2022

   

   tiran

   on Mar 29, 2022

   Member

   More actions

   I figured out how to implement copy(). dup() does not work as expected, but accept() on an AF_ALG client socket creates an independent copy.
8. ezio-melotti

   transferred this issue fromon Apr 10, 2022
9. gpshead commented on Feb 6, 2023

   

   gpshead

   on Feb 6, 2023

   MemberAuthor

   More actions

   Playing with your draft PR branch and testing things out... the performance of the Linux kernel crypto API for hash algorithms is several times slower on a modern AMD Zen, regardless of data size.

   I realize it probably gets a tad better in limited situations where sendfile or splice can used, but it doesn't seem like a useful API on modern mainstream hardware. It could still make sense on low performance things that have non-userspace accelerator hardware. But this doesn't leave much motivation to implement code using it.

   This one sounds like it could be a bit faster https://github.com/cryptodev-linux/cryptodev-linux (and is what would be needed for this on FreeBSD and OpenBSD), but is not in mainline in Linux.
10. arhadthedev commented on Feb 6, 2023

    

    arhadthedev

    on Feb 6, 2023

    Member

    More actions

    Playing with your draft PR branch and testing things out... the performance of the Linux kernel crypto API for hash algorithms is several times slower on a modern AMD Zen, regardless of data size.

    Probably, this should be added into the documentation if this issue gets rejected. Otherwise, there will be another attempts to introduce CryptoAPI, and one of them can even accidentally slip through.
11. gpshead commented on May 20, 2023

    

    gpshead

    on May 20, 2023

    MemberAuthor

    More actions

    Probably, this should be added into the documentation if this issue gets rejected. Otherwise, there will be another attempts to introduce CryptoAPI, and one of them can even accidentally slip through.

    I'm not worried about that. This issue existing in our issue tracker is sufficient documentation.
12. gpshead

    closed this as not plannedon May 20, 2023