use both "for in" and "ElementTree.remove" has a index bug

@scoder

BPO	41892
Nosy	@scoder, @ericvsmith, @ned-deily, @serhiy-storchaka, @miss-islington
PRs	bpo-41892: Clarify an example in the ElementTree docs #22464 bpo-41892: Clarify an example in the ElementTree docs #22546 [3.8] bpo-41892: Clarify that an example in the ElementTree docs explicitly avoids modifying an XML tree while iterating over it. (GH-22464) #22553 [3.9] bpo-41892: Clarify that an example in the ElementTree docs explicitly avoids modifying an XML tree while iterating over it. (GH-22464) #22554

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-05.04:22:13.395>
created_at = <Date 2020-09-30.14:20:25.049>
labels = ['type-bug', '3.8', '3.9', '3.10', 'expert-XML', 'invalid', 'docs']
title = 'use both "for in" and "ElementTree.remove" has a index bug'
updated_at = <Date 2020-10-05.04:22:13.395>
user = 'https://bugs.python.org/WoodyWoo'

bugs.python.org fields:

activity = <Date 2020-10-05.04:22:13.395>
actor = 'scoder'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-10-05.04:22:13.395>
closer = 'scoder'
components = ['Documentation', 'XML']
creation = <Date 2020-09-30.14:20:25.049>
creator = 'WoodyWoo'
dependencies = []
files = []
hgrepos = []
issue_num = 41892
keywords = ['patch']
message_count = 18.0
messages = ['377698', '377699', '377700', '377701', '377702', '377703', '377704', '377705', '377706', '377707', '377724', '377732', '377756', '377764', '377982', '377984', '377989', '377992']
nosy_count = 8.0
nosy_names = ['scoder', 'eric.smith', 'ned.deily', 'eli.bendersky', 'docs@python', 'serhiy.storchaka', 'miss-islington', 'WoodyWoo']
pr_nums = ['22464', '22546', '22553', '22554']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41892'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

#Just run it in Python v3.8.6 of win7 32bit 
import xml.etree.ElementTree as ET 
xmlstr='''<root>
<a no="1"/>
<b/>
<c/>
</root>'''
etroot = ET.fromstring(xmlstr)
for ELEMchild in etroot:
    if ELEMchild.get("no") == "1" :
        etroot.remove(ELEMchild)       #so far so good
    print (ELEMchild.tag)
    #It should be :  "b /n c" or "a /n b /n c",I can live with it both.
    #But it is :  "a /n c".
    #The index of ELEMchild should not +1 when there was a remove method worked on one of the before ELEMs.
    #I was forced to use while and if to control the index +1/+0.
    #BTW,ELEM has no method returning index in siblings, which is buging me too.

I assume that ElementTree doesn't support mutation while iterating.

However, the docs at https://docs.python.org/3/library/xml.etree.elementtree.html#modifying-an-xml-file show removing an item while iterating. It probably only works because the child being removed is the last one.

You will get the same behavior for lists:

>>> a = [1, 2, 3]
>>> for x in a:
...     if x == 1:
...         a.remove(x)
...     print(x)
... 
1
3

Lists are iterated by index. First you get an item at index 0, then at index 1, etc, to the end of the list. Initially the list is [1, 2, 3]. After removing 1 at the first iteration it becomes [2, 3], and at the next iteration you get an item at index 1 which is 3.

I think the only action here is to improve the documentation. That example is especially problematic.

@scoder

@eric.smith @scoder @serhiy.storchaka Thank U all.
I get what to do,and still think the "for in" structure should rebuilding.
All three methods:

import xml.etree.ElementTree as ET 
xmlstr=\
r'''<?xml version="1.0"?>
<data>
    <country name="Liechtenstein">
        <rank updated="yes">2</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank updated="yes">5</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
    <country name="Panama1">
        <rank updated="yes">69</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
    <country name="Panama2">
        <rank updated="yes">69</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
</data>'''
print(xmlstr)

#orginal code
root = ET.fromstring(xmlstr)
for country in root.findall('country'):
    rank = int(country.find('rank').text)
    if rank > 50:
        root.remove(country)
print("___orginal___")
for country in root.findall('country'):
        print (country.get("name"))
print("^^^orginal^^^^\n")

#wrong code in my mind
root = ET.fromstring(xmlstr)
for country in root:
    rank = int(country.find('rank').text)
    if rank > 50:
        root.remove(country)
print("___bad___")
for country in root.findall('country'):
        print (country.get("name"))
print("^^^bad^^^^\n")

#my code
root = ET.fromstring(xmlstr)
index=0
count=len(root.findall("./*"))
while index <count:
    rank = int (root[index].find("rank").text)
    if rank>50:
        root.remove(root[index])
        index=index+0
        count=count-1  # avoid index err
        continue
    index=index+1
print("___new___")
for country in root.findall('country'):
        print (country.get("name"))
print("^^^new^^^^\n")

Could I say the mutable sequence containing not the object but the pointer like C++.
So they can changed in def functions.

I can't say how ElementTree works without more checking, but this solution cannot work in general. Given a pointer to an object that's in a list, how would you get to the next item? Say the parent list-like object has a C array of pointers to the objects it contains, and removing one of the objects re-shuffles that list. How would keeping a pointer to the current object help you?

In any event, I don't think this behavior is going to change.

New changeset 40db798 by scoder in branch 'master':
bpo-41892: Clarify that an example in the ElementTree docs explicitly avoids modifying an XML tree while iterating over it. (GH-22464)
40db798

New changeset 6bd058e by Miss Skeleton (bot) in branch '3.8':
bpo-41892: Clarify that an example in the ElementTree docs explicitly avoids modifying an XML tree while iterating over it. (GH-22464)
6bd058e

New changeset d571924 by Miss Skeleton (bot) in branch '3.9':
bpo-41892: Clarify that an example in the ElementTree docs explicitly avoids modifying an XML tree while iterating over it. (GH-22464) (GH-22554)
d571924

Closing since this works as designed.

Users are responsible for avoiding concurrent tree modifications during iteration.

Navigation Menu

Repository navigation

use both "for in" and "ElementTree.remove" has a index bug #86058

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

use both "for in" and "ElementTree.remove" has a index bug #86058

Description

Activity

WoodyWoo commented on Sep 30, 2020

ericvsmith commented on Sep 30, 2020

serhiy-storchaka commented on Sep 30, 2020

ericvsmith commented on Sep 30, 2020

8 remaining items

WoodyWoo commented on Oct 1, 2020

WoodyWoo commented on Oct 1, 2020

ericvsmith commented on Oct 1, 2020

ned-deily commented on Oct 4, 2020

miss-islington commented on Oct 4, 2020

ned-deily commented on Oct 4, 2020

scoder commented on Oct 5, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions