Repository navigation
ElementTree incorrectly refuses to write attributes without namespaces when default_namespace is used #61290
Description
| BPO | 17088 |
|---|---|
| Nosy | @rhettinger, @scoder, @silverbacknet, @vadmium, @SpecLad, @mthuurne |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-01-31.02:35:07.204>
labels = ['expert-XML', 'type-bug', '3.10', '3.11']
title = 'ElementTree incorrectly refuses to write attributes without namespaces when default_namespace is used'
updated_at = <Date 2021-07-19.12:58:02.922>
user = 'https://github.com/SilverbackNet'bugs.python.org fields:
activity = <Date 2021-07-19.12:58:02.922>
actor = 'scoder'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['XML']
creation = <Date 2013-01-31.02:35:07.204>
creator = 'silverbacknet'
dependencies = []
files = ['32981', '33125']
hgrepos = []
issue_num = 17088
keywords = ['patch']
message_count = 17.0
messages = ['181005', '205281', '205951', '206156', '206186', '206217', '209424', '216061', '216067', '261785', '304120', '331390', '331432', '354101', '365760', '397793', '397794']
nosy_count = 14.0
nosy_names = ['rhettinger', 'scoder', 'eli.bendersky', 'wiml', 'silverbacknet', 'martin.panter', 'SpecLad', 'gene_wood', 'marczellm', 'Garrett Birkel', 'Rafael Ascensao', 'mthuurne', 'carlschaefer', 'dtluttik']
pr_nums = ['11050']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17088'
versions = ['Python 3.10', 'Python 3.11']
Activity
SilverbackNet commentedon Jan 31, 2013
ET reads a default-namespaced (xmnls="whatever") file correctly but won't write it back out.
The error given is:
ValueError: cannot use non-qualified names with default_namespace option
The XML reference is reasonably clear on this:
http://www.w3.org/TR/REC-xml-names/#defaulting
"Default namespace declarations do not apply directly to attribute names;"
"The namespace name for an unprefixed attribute name always has no value."
Therefore, it is not an error to write non-qualified _attribute_ names with a default namespace; they're just considered un-namespaced anyway. The trivial case where a file is read in with a default namespace and written out with the same one should make it obvious:
Yet this will fail with the error above. By leaving off default_namespace, every element is pointlessly prefixed by 'svg:' in the resulting file, but it does work.
wiml commentedon Dec 5, 2013
I have run into this a few times although it is only recently that I've convinced myself I understood the XML namespace spec well enough to know what the right behavior was. (I came to the same interpretation as silverbacknet.)
I have attached a patch which I believe fixes (and tests) the problem.
wiml commentedon Dec 12, 2013
FWIW: I noticed that my patch has a bug due to sharing the cache dict between element names and attribute names, although I think this is unlikely to crop up very often in practice. I'll submit a better patch if/when I get the time to put one together.
wiml commentedon Dec 13, 2013
Here's an improved patch (and improved testcase).
It's a little more intrusive than the last patch because when a default namespace is being used, two distinct qname caches must be made.
scoder commentedon Dec 14, 2013
Note that the option is called "default_namespace", not "default_namespace_prefix". Could you try passing the namespace URI instead?
wiml commentedon Dec 15, 2013
Yes, the problem occurs regardless of whether the default_namespace parameter is the correct SVG namespace URI --- it's the fact of requesting a default namespace at all that exposes the bug.
wiml commentedon Jan 27, 2014
Ping
genewood commentedon Apr 14, 2014
One workaround to this is described here : http://stackoverflow.com/a/4999510/168874
It involves prefixing all of the elements with the namespace like this :
scoder commentedon Apr 14, 2014
@gene_wood: that's unrelated. This ticket is about attributes being rejected incorrectly.
Fixing the example of the OP:
So, it works without namespace defaulting and fails with an incorrect error when a default namespace is provided. Clearly a bug.
Regarding the proposed patch: it looks like the right thing to do in general, but it has a relatively high code impact. I would prefer a patch with lower churn. One thing that could be tried is to use only one tag cache dict and extend the key from the plain tag to (tag, is_attribute). Might have a performance impact on the already slow serialiser, though. In any case, both approaches are quite wasteful, because they duplicate the entire namespace-prefix mapping just because there might be a single namespace that behaves differently for atributes. An alternative could be to split the *value* of the mapping in two: (element_prefix, attribute_prefix). This would keep the overhead at serialisation low, with only slightly more work when building the mapping. At first sight, I like that idea better.
This code returns a list in one case and a set-like view in another (Py3):
+ if default_namespace:
+ prefixes_list = [ (default_namespace, "") ]
+ prefixes_list.extend(namespaces.items())
+ else:
+ prefixes_list = namespaces.items()
I can't see the need for this change. Why can't the default namespace be stored in the namespaces dict right from the start, as it was before?
As a minor nitpick, this lambda based sort key:
is better expressed using operator.itemgetter(1).
I'd also rename the "defaultable" flag to "is_attribute" and pass it as keyword argument (bare boolean parameters are unreadable in function calls).
Given the impact of this change, I'd also suggest not applying it to Py2.x anymore.
GarrettBirkel commentedon Mar 14, 2016
Just hit up against this namespace bug. Has this patch been abandoned??
RafaelAscensao commentedon Oct 11, 2017
what's the status on this?
mthuurne commentedon Dec 8, 2018
I was working on what I thought would be an elegant solution to this problem: for non-qualified attributes, add the element's namespace before accessing the cache and strip the namespace prefix after accessing the cache if it's equal to the element's prefix.
However, this approach doesn't work: even though non-qualified attributes will be processed like they are the element's namespace, they are considered to have no namespace. This means <ns:x a="1" ns:a="2"/> is considered valid XML, even though it effectively defines the same attribute twice.
https://www.w3.org/TR/REC-xml-names/#uniqAttrs
In my opinion the spec made a silly choice here, but that's probably not something that can fixed anymore.
I haven't decided yet whether I'll make another attempt at fixing this issue. In any case, I hope this tale of caution benefits someone.
mthuurne commentedon Dec 9, 2018
I think I have a good solution now, see the pull request for details. It does touch a lot of code, but I split all the changes into small consistent units, so it should be easier to verify whether they are correct.
4 remaining items
scoder commentedon Jul 19, 2021
The obvious work-around is to not use a default namespace. The result is just a visual difference, not a semantic one.
If someone wants to continue with the existing PR, I'll try to free some time to review any improvements.
couling commentedon Jan 5, 2023
@scoder The result is not just visual.
Strictly speaking prefixes have no semantic effect in XML. But unfortunately this has not stopped developers erroneously relying on the prefix. Many libraries including xml.dom expose the prefix. ElementTree is a little unusual in the software world for not exposing it. Unfortunetly people have used it.
This issue can cause problems when interfacing with other systems despite the XML standard. Yes, you can argue the other system is broken, but that's of little use to a developer with no ability to change the other side of an interface.
The
register_namespaceworkaround dodges this problem, but is painful for having to set something globally.Using the default namespace attributes is so common that it would be fair to say that
default_namespace=has never worked for XML documents with attributes.To my mind a better solution would be to depreciate
default_namespaceentirely and replace it with a newprefixesargument supplying a dictionary of{"Namespace URI": "prefix"}(or visa versa) that take priority over the globally registered one.One implementation decision would be: when handling namespaces not in
prefixesthe solution could either:prefixesnot set then, and only then, the globally registered namespaces would take effect.Since this is a new param, it would be backward compatible.
pythongh-61290: Fix ElementTree's serialization of attr names when de…
serhiy-storchaka commentedon Aug 31, 2026
PR #156747 fixes this.
A default namespace declaration does not apply to attribute names (Namespaces in XML, 6.2), so an unqualified attribute name can be written as is. A qualified attribute name always needs a prefix, even if it is in the default namespace, so its namespace is declared twice:
lxml serializes it the same way.
@couling, the prefixes argument you proposed is a separate feature; this only removes the error.
Merge remote-tracking branch 'upstream/main' into pythongh-61290-defa…
gh-61290: Fix serializing attributes with the default_namespace option (
pythongh-61290: Fix serializing attributes with the default_namespace…