Repository navigation
Allow ssl certificates to be specified from memory rather than files. #60691
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2012-11-16.15:10:15.811>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7', '3.9']
title = 'Allow ssl certificates to be specified from memory rather than files.'
updated_at = <Date 2020-03-01.22:19:26.571>
user = 'https://github.com/kristjanvalur'bugs.python.org fields:
activity = <Date 2020-03-01.22:19:26.571>
actor = 'kristjan.jonsson'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['SSL']
creation = <Date 2012-11-16.15:10:15.811>
creator = 'kristjan.jonsson'
dependencies = []
files = ['28002', '28866', '30707', '30718', '32612', '44092']
hgrepos = []
issue_num = 16487
keywords = ['patch']
message_count = 67.0
messages = ['175691', '175692', '175701', '175709', '175741', '175779', '175806', '175809', '180734', '180736', '180737', '190997', '191018', '191828', '191838', '191847', '191859', '191860', '191862', '191868', '191922', '191923', '191924', '191936', '191937', '191938', '191941', '191942', '191957', '191970', '202800', '202828', '202829', '202832', '202833', '246488', '246489', '246492', '246494', '246495', '246496', '246555', '272585', '275288', '277173', '281457', '293908', '297068', '303514', '307153', '307154', '307296', '307298', '307299', '307320', '307322', '307357', '307372', '307373', '307377', '307380', '307681', '309841', '324686', '362692', '363074', '363101']
nosy_count = 13.0
nosy_names = ['jcea', 'orsenthil', 'pitrou', 'kristjan.jonsson', 'giampaolo.rodola', 'christian.heimes', 'njs', 'asvetlov', 'jgehrcke', 'brandon-rhodes', 'dstufft', 'martius', 'Lord Anton Hvornum']
pr_nums = ['2449']
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16487'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
kristjanvalur commentedon Nov 16, 2012
The _ssl module (and indeed the openssl lib) relies heaviliy on actual filesystem locations to load certificates. A client or a server may not want to rely on physical filesystem locations to load certificates for authentication or verification. Physical disc files are cumbersome and present a management burden in the presence of multiple processes.
This patch adds extensions to the _ssl.c file which allows certificates, keys and certification chains to be provided by file contents, rather than file name.
The ctx.load_cert_chain and ctx.load_verify_locations take additional arguments to specify the data on this form.
the ssl.wrap_socket does not add arguments, rather the function is polymorphic in that the conents of the certfil/keyfile are examined and treated as file-data if beginning with -----BEGIN. the ca_certs is similarly treated as a list of file contents, if it is a list, (rather than a string)
This patch is the result of work at CCP for deploying ssl clients and servers in an isolated environment without having to resort to temporary disk files.
kristjanvalur commentedon Nov 16, 2012
(the patch contains a local change to set the location of the 'external' dir, please disregard this.
pitrou commentedon Nov 16, 2012
Can you upload a cleaned up patch then? :)
kristjanvalur commentedon Nov 16, 2012
Can't right now. It's only relevant for pcbuild anyway so you can test it for Unix if you want without fear. Don't worry, I always give my patches a cleanup before committing them.
Meanwhile, I'd welcome comments on the substance.
pitrou commentedon Nov 17, 2012
I don't want to review a patch if you tell me that it has problematic stuff in it. Please first upload a cleaned up patch.
pitrou commentedon Nov 17, 2012
By the way, without looking too much at the patch, I think it would be cleaner to let the certificate functions accept file-like objects, rather than adding keydata/certdata arguments.
kristjanvalur commentedon Nov 17, 2012
The please don't. I'm actually growing tired of the way you seem to always latch onto every submission I make almost with hostility. I will find someone else to look this over.
pitrou commentedon Nov 17, 2012
I fully realize my reply was blunt. But perhaps you could try to
understand that people care about the quality of the codebase and it is
why they care about reviewing patches before letting them in. Asking for
reviews of patches that you *know* contain unneeded cruft is rude.
Disregarding their comments and saying "I'll ask someone else because I
don't want to deal with you anymore" is even worse.
This certainly happens often to you, because (IME) you are often pushy
with your patches. So, yes, you must expect a certain amount of
defensive replies when being so. Just be prepared to deal with them.
jcea commentedon Jan 27, 2013
Kristján, are you pursuing this yet?. Can we move on?
[-]Allow ssl certificates to be speficfied from memory rather than files.[/-][+]Allow ssl certificates to be specified from memory rather than files.[/+]kristjanvalur commentedon Jan 27, 2013
I wasn't pushing it anymore because there seemed to be no interest and Antoine attitude made me lose interest in contributing this. No one liked to comment on my approach except Antoine and I had grown tired of his persistent negativity and aloof manners. I might as well say that here that it is this kind of feedback that has turned me away from contributing to the project on several occations. At CCP we have a heavily patched version of python that we have modified throug time to deal with real world projects. I (and my company) genuinely like to contribute some of those improvements back to the trunk and regularly make efforts to do so (e.g. by taking the time and effort to port to python 3) but often they are met with with almost hostile negativity. I hope, for the sake of Python, that I am the only contributor meeting this kind of attitude.
Antoine, please take the time to read what I posted here again. I had a working patch, used in production. I uploaded, it, but when I got back home realized that an unrelated change was part of it. Unable to change it from there I asked politely that you disregard that "unrelated" change while looking at the _substance_ of patch. As you know, I am a committer and will polish things before checking them in, as anyone does.
How is that rude?
The patch still stands. I on a poor internet in a Chinese hotel and will attempt to download and edit the patch so that the irrelevant change to the PC build solution should no longer need to hinder anyone from reading it.
56 remaining items
pitrou commentedon Dec 1, 2017
Le 01/12/2017 à 10:47, Christian Heimes a écrit :
Yes, that's what I meant.
orsenthil commentedon Dec 5, 2017
Hi Cristian,
I think this (more C code) is the primary and the *only* negative point against the current patch. And that seems necessary for the feature specific to OpenSSL.
Not sure if you looked at the latest version (https://github.com/python/cpython/pull/2449/files) recently.
The current patch does not deviate in-principle from the PEP-543.
It maintains the same API arguments
SSLContext.load_cert_chain(certfile, keyfile=None, password=None)We expect the migration of ssl module to newer ABC of PEP-543 not be one-to-one. We could foresee this API living. (And we haven't deprecated this API).
The patch provides the feature along with plenty of tests that PEP-543 talks about (Loading of certs from memory).
Has an implementation refactorable for OpenSSL-specific TLS backend (as one of the provider), that again will be useful to PEP-543 implementation.
These are the benefits in my opinion.
PEP-543 is important and seems like a *major effort*. The current patch might still be valuable and perhaps might be useful towards PEP-543 implementation. It deals only with certificates only.
tiran commentedon Jan 12, 2018
Senthil,
I'm not a fan of PR 2449 because it provides yet another way to load certificates and keys from memory. It's a clever idea to use MemoryBIO here. But the approach is *not* compatible with PEP-543. The PEP requires an API that can turn a memory blob into a series of certificate objects. PR 2449 doesn't enable memory -> certificate. The new API is OpenSSL specific and restricted to PKCS#8 key in PEM/DER encoding. PEP-543 is extensible for PKCS#11, engine and HSM support. PR 2449 is not.
There are other issues with PR 2449. For example it's missing several GIL releases calls. The password callback doesn't look correct in some places.
#5162 provides a PEP-543-compatible implementation (plus additions from pending PR):
# get_ca_certs() doesn't return ssl.Certificates yet >>> ctx.get_ca_certs() [{'subject': ((('countryName', 'XY'),), (('organizationName', 'Python Software Foundation CA'),), (('commonName', 'our-ca-server'),)), 'issuer': ((('countryName', 'XY'),), (('organizationName', 'Python Software Foundation CA'),), (('commonName', 'our-ca-server'),)), 'version': 3, 'serialNumber': 'B09264B1F2DA21D0', 'notBefore': 'Jan 4 19:47:07 2013 GMT', 'notAfter': 'Jan 2 19:47:07 2023 GMT'}]jgehrcke commentedon Sep 6, 2018
Thanks for the discussion.
Since I tried to join the efforts here in 2016 two years ago I was (and still am) enthusiastic, and willing to invest quite a bit of energy. Still, we have missed the 3.6 and 3.7 releases to change something about the fact that there is no simple way to load certificate data from memory rather than files.
I really appreciate the work that as been done on https://www.python.org/dev/peps/pep-0543/. It's a truly great effort. But I think the fact that we did not land a solution since 2012 is somewhat an indicator showing that building smaller chunks and showing the willingness to iterate on a solution would have led to a more tangible outcome, something that I could use right now when building Python applications (how great would that be?). Honestly, the small API changes that were discussed here for years and that I think I implemented do not seem problematic to me from a maintenance and deprecation perspective. Yes, PEP-543 proposes a more idealistic way of doing things. But I do not see how implementing PEP-543 would have been slowed down or negatively affected in any way by the last patch I proposed.
#5162 looks very big, and C is not as much my comfort zone as is Python. Still, please let me know if you would like me to contribute!
By the way, I would have _really_ liked to see the following feedback around the first time I submitted the patch:
I remember that this aspect of writing the patch consumed quite a bit of my time and I operated with care, knowing the risks. I very much would have appreciated specific feedback on this. To learn, and to actually address problems.
Jan-Philip
LordAntonHvornum commentedon Feb 26, 2020
I agree with Jan-Philip Gehrcke, would have been nice to have had this.
Pretty pissed reading through this ancient issue.
In regards to how people treat other volunteers: We're all working for free, and I think expectations from people with authoritative powers must have room for accepting a initially lower contribution standard than they themselves claim to produce. Mainly to not crush the will to help and contibute, but also because the attitude could be a lot better towards fellow programmers. It would also allow for a more iterative process in achieving higher standards. Otherwise you limit yourself to a very select few in the contribution process. Not to mention a lot of "grunt work" could be done by people having an idea, but not nessecarily the same "academic background" in terms of producing perfect code.
I've seen this across numerous Open Source projects, and quite frankly it deter me from ever contributing. PoC code shouldn't be seen as garbage, but a first step in creating new fresh things. Someone with more knowledge should and could step in and clean up things that are deamed "unfit" for production or help said contributor to understanding why certain things needs to be improved, not just blatently say patches needs to be improved.
This whole "It ain't perfect, we don't touch it" is elitism at it's finest. Stop it, and have a meaningful discussion instead.
jgehrcke commentedon Mar 1, 2020
I am not too attached to "my" patch, but because I love Python I really would like us to land on a solution.
Christian, I wonder: did we make progress on this front? Will the stdlib in the 3.9 release make it easy to load cert and key material from memory?
Thanks Anton for your words, truly appreciated. I do not want to sound too negative, but I would like to confirm that emotionally this issue here was a bit of an unpleasant experience for me, and it certainly inhibited my intentions, energy, motivation to (try to) contribute more. Thanks, though, to all supporters here.
kristjanvalur commentedon Mar 1, 2020
I gave up contributing a long time ago now because it was too emotionally exhausting to me. This issue was one that helped tip the scales. I hope things have become easier now because good projects like Python need the enthusiasm and spirit of volunteer contributors. Good luck.
alonbl commentedon Apr 30, 2022
Hello,
Any news? it is important to be able to use private key from external devices and it fairly trivial to let openssl implement this. In future it can be extended but for now, please support it with minimal effort if using the openssl way.
Thanks,
arhadthedev commentedon Feb 14, 2023
Closing; see #2449 (comment) for explanation.
lapcchan commentedon Mar 14, 2024
@jgehrcke thanks for your patch and although it's not getting merge but it does help us to avoid writing keyfile on disk.
it's seems that workaround patches are not welcome so i am expecting this feature request will remains open for another 10yrs lol
GunGunGun commentedon May 10, 2024
Kinda demoralising to know this won't be improved, but well up to some point Python really needs to move on from OpenSSL and switch to RustTLS or something like, OpenSSL isn't that open, it's so restricted that's why it's getting hit hard by TLSFingerprint, no way to workaround btw.