cgi module cannot handle POST with multipart/form-data in 3.x

@warsaw

BPO	4953
Nosy	@warsaw, @birkenfeld, @amauryfa, @vstinner, @merwok, @bitdancer, @florentx, @PierreQuentel
Files	full_source_and_error.zip: full source and error show opsuper.pl: Perl code for process post files from form unittest.zip: cgi.FieldStorage unittest for Multipart form-data and associated files. http.zip: Module cgi_new.py and tests cgitest-python3.py cgi_tests.zip: cgi_test.py and associated files adder.html: uses get, works adderpost.html: just hanged get to post, makes adder.cgi hang localCGIServer.py: my wrapper around http.server to handle localhost adder.cgi: action script used by adder.html. adderpost.html, hangs with adderpost.html

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-01-14.13:12:27.362>
created_at = <Date 2009-01-15.09:02:45.958>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'cgi module cannot handle POST with multipart/form-data in 3.x'
updated_at = <Date 2011-01-15.01:19:29.678>
user = 'https://bugs.python.org/oopos'

bugs.python.org fields:

activity = <Date 2011-01-15.01:19:29.678>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-01-14.13:12:27.362>
closer = 'vstinner'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2009-01-15.09:02:45.958>
creator = 'oopos'
dependencies = []
files = ['12753', '12768', '14235', '20217', '20270', '20385', '20393', '20394', '20395', '20396']
hgrepos = []
issue_num = 4953
keywords = ['patch']
message_count = 130.0
messages = ['79892', '79893', '79894', '79896', '79898', '79901', '79939', '79981', '80000', '88960', '88962', '89112', '91444', '91449', '91711', '91741', '95288', '95292', '95330', '107959', '108221', '121864', '125035', '125065', '125086', '125088', '125100', '125105', '125106', '125108', '125114', '125152', '125153', '125158', '125159', '125178', '125181', '125201', '125217', '125235', '125237', '125241', '125402', '125403', '125410', '125419', '125426', '125428', '125474', '125481', '125483', '125501', '125524', '125533', '125543', '125546', '125556', '125557', '125558', '125563', '125570', '125583', '125629', '125633', '125637', '125698', '125839', '125840', '125885', '125886', '125892', '125893', '125900', '125901', '125921', '125926', '125928', '125930', '125931', '125935', '125952', '125992', '125993', '125994', '126035', '126060', '126062', '126065', '126066', '126075', '126117', '126124', '126140', '126145', '126152', '126160', '126161', '126162', '126164', '126165', '126167', '126173', '126175', '126199', '126205', '126207', '126210', '126212', '126214', '126215', '126219', '126222', '126232', '126233', '126242', '126249', '126251', '126253', '126256', '126257', '126262', '126264', '126266', '126267', '126268', '126269', '126289', '126293', '126301', '126307']
nosy_count = 17.0
nosy_names = ['barry', 'georg.brandl', 'amaury.forgeotdarc', 'ggenellina', 'vstinner', 'andyharrington', 'eric.araujo', 'grahamd', 'v+python', 'r.david.murray', 'oopos', 'tcourbon', 'tobias', 'flox', 'pebbe', 'quentel', 'erob']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4953'
versions = ['Python 3.2']

Python 3.0 (r30:67507, Dec 3 2008, 20:14:27) [MSC v.1500 32 bit
(Intel)] on win32
---------------------------------------------
Hi,I user Python!

UnicodeDecodeError Python 3.0: G:\cgi\python\python.exe
Thu Jan 15 16:46:34 2009

A problem occurred in a Python script. Here is the sequence of function
calls leading up to the error, in the order they occurred.
G:\webserver\xampp\cgi-bin\testupload.py in ()
107
108 # get form
109 opsform = cgi.FieldStorage()
110
111 print ("<br>","form-data:","<br>",opsform,"<br>")
opsform undefined, cgi = <module 'cgi' from 'G:\cgi\python\lib\cgi.py'>,
cgi.FieldStorage = <class 'cgi.FieldStorage'>
G:\cgi\python\lib\cgi.py in __init__(self=FieldStorage(None, None, []),
fp=None, headers={'content-length': '671631', 'content-type':
'multipart/form-data;
boundary=---------------------------9699301019407'}, outerboundary='',
environ=<os._Environ object at 0x00C90C50>, keep_blank_values=0,
strict_parsing=0)
477 self.read_urlencoded()
478 elif ctype[:10] == 'multipart/':
479 self.read_multi(environ, keep_blank_values,
strict_parsing)
480 else:
481 self.read_single()
self = FieldStorage(None, None, []), self.read_multi = <bound method
FieldStorage.read_multi of FieldStorage(None, None, [])>, environ =
<os._Environ object at 0x00C90C50>, keep_blank_values = 0,
strict_parsing = 0
G:\cgi\python\lib\cgi.py in read_multi(self=FieldStorage(None, None,
[]), environ=<os._Environ object at 0x00C90C50>, keep_blank_values=0,
strict_parsing=0)
597 # Create bogus content-type header for proper multipart
parsing
598 parser.feed('Content-Type: %s; boundary=%s\r\n\r\n' %
(self.type, ib))
599 parser.feed(self.fp.read())
600 full_msg = parser.close()
601 # Get subparts
parser = <email.feedparser.FeedParser object at 0x00DD5650>, parser.feed
= <bound method FeedParser.feed of <email.feedparser.FeedParser object
at 0x00DD5650>>, self = FieldStorage(None, None, []), self.fp =
<io.TextIOWrapper object at 0x00BE3FB0>, self.fp.read = <bound method
TextIOWrapper.read of <io.TextIOWrapper object at 0x00BE3FB0>>
G:\cgi\python\lib\io.py in read(self=<io.TextIOWrapper object at
0x00BE3FB0>, n=-1)
1722 # Read everything.
1723 result = (self._get_decoded_chars() +
1724 decoder.decode(self.buffer.read(), final=True))
1725 self._set_decoded_chars('')
1726 self._snapshot = None
decoder = <encodings.gbk.IncrementalDecoder object at 0x00DB7AB0>,
decoder.decode = <built-in method decode of IncrementalDecoder object at
0x00DB7AB0>, self = <io.TextIOWrapper object at 0x00BE3FB0>, self.buffer
= <io.BufferedReader object at 0x00BE3F90>, self.buffer.read = <bound
method BufferedReader.read of <io.BufferedReader object at 0x00BE3F90>>,
final undefined

UnicodeDecodeError: 'gbk' codec can't decode bytes in position 157-158:
illegal multibyte sequence
args = ('gbk',
b'-----------------------------9699301019407\r\n...-----------------------------9699301019407--\r\n',
157, 159, 'illegal multibyte sequence')
encoding = 'gbk'
end = 159
object =
b'-----------------------------9699301019407\r\n...-----------------------------9699301019407--\r\n'
reason = 'illegal multibyte sequence'
start = 157
with_traceback = <built-in method with_traceback of
UnicodeDecodeError object at 0x00DB7BF0>

l:\tmp_dir\tmpxyeojf.html contains the description of this error.

-------------------------------------------
Hi,
I am newbie for python under the windows.
I find that the cgi module always work wrong for the binary files to upload.
I find that it cannot auto to discern the files' mode and alway use the
default mode : 'TEXT'.
So I want to change the sys.stdin 's mode to BINARY to support the
binary files.
I got this way:
import msvcrt,os
msvcrt.setmode(0,os.O_BINARY) # for stdin
msvcrt.setmode(1,os.O_BINARY) # for stdout
but it isn't work,too.
I know use C progam language can use this function:
freopen("somefilename","mode","stdin or stdout") to redirect the file
flow.
Can every one help me ?

Best Regards
oopos

I

Does it work if you change your script like this:
opsform = cgi.FieldStorage(open(sys.stdin.fileno(), 'rb'))

To Amaury Forgeot d'Arc :

Thank you.

That error have sloved with your way:
[quote]Does it work if you change your script like this:
opsform = cgi.FieldStorage(open(sys.stdin.fileno(), 'rb'))[/quote]

Now,The new problem come out:
[code] 97 """Push some new data into this object."""

98 # Handle any previous leftovers

99 data, self._partial = self._partial + data, ''

100 # Crack into lines, but preserve the newlines on the end
of each

101 parts = NLCRE_crack.split(data)

data = b'-----------------------------7d91f41a302f4
\nCo...\x0e\x0f\x0c\x10\x17\x14\x18\x18\x17\x14\x16\x16', self =
<email.feedparser.BufferedSubFile object at 0x00DD5270>, self._partial
= ''

TypeError: Can't convert 'bytes' object to str implicitly
[/code]

I find that the CGI LIB didn't use bytes flow, it always use string
flow.

More info in the attch file:

OK, another try. Please replace the previous version with these 3 lines:

  encoding = os.environ.get('HTTP_TRANSFER_ENCODING')
  stdin = open(sys.stdin.fileno(), 'r', encoding=encoding)
  opsform = cgi.FieldStorage(stdin)

Thank you for time.
Now,I try with you saied.Bu it is taken wrong as before.

See the files:

Thanks for the test case. I reproduced it easily.
There is indeed a real problem in CGI streams.

The first thing to do is to start python with the -u option (add it to
the end of the first #! line), so that stdin yields bytes instead of
unicode chars, and \r\n are not translated on Windows.

Even then, I noticed that in the multipart/form-data section, text
fields are utf-8 encoded, but the file content is raw binary.
(FWIW, I use Firefox and Apache on Windows)
No encoding seems to be specified, neither in the content, nor in the
environment (no HTTP_TRANSFER_ENCODING)

And of course, the email.parser.FeedParser object used to parse it
accepts only unicode, not bytes.
Help needed.

An attempt to more accurately describe the issue, to attract more
knowledgeable people, I hope...

Hehe.
I only want to use python to slove the upload files instead of PHP.but
it is hard to me or I have no much time to leran it.
Now,I learn Perl quickly and use it upload files, it works ok.

Thank you . I will take more time to learn Python language well.
Best Regards!

This is my code: (Perl Language)

You should stick to Python 2.6 (or even 2.5) for web programming - 3.0
is not mature enough. I thought this was a feasibility study on porting
an existing application to Python 3.0 -- not your first steps in the
language.

r87996+r87997 adds encoding and errors argument to parse_qs() and parse_qsl() of urllib.parse. It is needed to decoded correctly %XX syntax in cgi.

r87998 is the patch on the cgi module.

Changes with cgi_32.patch:

Use TextIOWrapper instead of TextIOBase, because TextIOBase has no buffer
attribute
typo in a docstring: "it must must" => "must"
(docstring) default: sys.stdin => default: sys.stdin.buffer
PEP-8: hasattr(a,b) => hasattr(a, b) (same for isinstance) and
"encoding = 'utf-8'" => "encoding='utf-8'" (in the argument list)
"xxx.decode(...) # str": remove useless # str comment. decode() always give
unicode in Python 3 (same change for ".encode() # bytes")
Rename "next" variables to "next_boundary" because next is a builtin
function in Python 3 (unrelated change).
FieldStorage.innerboundary and FieldStorage.outerboundary are bytes objects:
encode innerboundary in the constructor, and raise an error if outerboundary
is not a bytes object
Rename _use_bytes to _binary_file
isinstance(bytes) test: write the type, not the value, in the error message
Replace line[:2] == b'--' by line.startswith(b'--'), and then replace
line.strip() by line.rstrip()
test_fieldstorage_multipart() uses ASCII (and specifiy the encoding to FieldStorage)
add FieldStorage.errors attribute: pass it to parse_qsl()
add errors attribute to FieldStorage: same default value than urllib.parse.unquote(): 'replace'
parse(): pass encoding argument to parse_qs()
FieldStorage: pass encoding and errors arguments to parse_qsl()

Because the patch on TextIOBase, it patched the docstring:
---
fp : file pointer; default: sys.stdin.buffer
(not used when the request method is GET)
Can be :
1. an instance of (a subclass of) TextIOWrapper, in this case it
must provide an attribute "buffer" = the binary layer that returns
bytes from its read() method, and preferably an attribute
"encoding" (defaults to latin-1)
2. an object whose read() and readline() methods return bytes
---
becomes
---
fp : file pointer; default: sys.stdin.buffer
(not used when the request method is GET)
Can be :
1. a TextIOWrapper object
2. an object whose read() and readline() methods return bytes
---

Replace "type(value) is type([])" is done in another commit: r87999.

I consider that the work on this issue is done, and so I close it. If I am wrong, explain why and repoen the issue.

Please test the cgi as much as possible before Python 3.2 final: reopen the issue if it doesn't work.

Oh, I forgot to credit the author(s): who wrote the patch?

Thanks a lot Victor !

I wrote the patch : Pierre Quentel (pierre.quentel@gmail.com) with many
inputs by Glenn Linderman

2011/1/14 STINNER Victor <report@bugs.python.org>

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

Oh, I forgot to credit the author(s): who wrote the patch?

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4953\>

TODO: Add more tests to test_cgi. What is the latest patch for test_cgi?

My latest patch for test_cgi is in cgi_32.patch

I will try to add more tests later

haypo>> What is the latest patch for test_cgi?
quentel> My latest patch for test_cgi is in cgi_32.patch

Ok, but cgi_32.patch doesn't add any test. I only adapt existing tests for your other changes.

I remove cgi_32.patch because it was commited.

Remove cgi_plus_tests.diff: it looks to be an old version of cgi_32.patch.

@r.david.murray: Did you write cgi_plus_tests.diff, or is it based on the work on Pierre Quentel?

Remove tmpy44zj7.html and tmpav1vve.html: a similar file is included in full_source_and_error.zip.

Victor: we normally leave the patch file that was committed attached to the issue for future reference.

The _plus_tests file was just the original patch plus the existing cgi tests adjusted to pass in bytes instead of strings to cgi, if I recall correctly. Nor original code of mine in either part :)

Thanks to Pierre for producing patch after patch and testing testing testing, and to Victor for committing it, as well as others that contributed in smaller ways, as I tried to. I look forward to 3.2 rc1 so I can discard all my temporary patched copies of cgi.py

Because I'm unable to read the whole history and analyze each file attached to this issue, I opened bpo-10911 to ask to write more tests for the cgi module.

Le vendredi 14 janvier 2011 à 19:11 +0000, R. David Murray a écrit :

Victor: we normally leave the patch file that was committed attached
to the issue for future reference.

Sorry, but there were too much files. I was trying to figure out if
there is something useful in the files.

The _plus_tests file was just the original patch plus the existing cgi
tests adjusted to pass in bytes instead of strings to cgi, if I recall
correctly. Nor original code of mine in either part :)

Ok.

Navigation Menu

Repository navigation

cgi module cannot handle POST with multipart/form-data in 3.x #49203

116 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

cgi module cannot handle POST with multipart/form-data in 3.x #49203

Description

Activity

oopos commented on Jan 15, 2009

amauryfa commented on Jan 15, 2009

oopos commented on Jan 15, 2009

amauryfa commented on Jan 15, 2009

oopos commented on Jan 15, 2009

amauryfa commented on Jan 15, 2009

ggenellina commented on Jan 16, 2009

oopos commented on Jan 16, 2009

ggenellina commented on Jan 17, 2009

116 remaining items

vstinner commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

PierreQuentel commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

PierreQuentel commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

bitdancer commented on Jan 14, 2011

vpython commented on Jan 14, 2011

vstinner commented on Jan 14, 2011

vstinner commented on Jan 15, 2011

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions