Skip to content

Commit 755ebea

Browse files
tim-onetim-one
committed
PyNode_AddChild(): Do aggressive over-allocation when the number of
children gets large, to avoid severe platform realloc() degeneration in extreme cases (like test_longexp). Bugfix candidate. This was doing extremely timid over-allocation, just rounding up to the nearest multiple of 3. Now so long as the number of children is <= 128, it rounds up to a multiple of 4 but via a much faster method. When the number of children exceeds 128, though, and more space is needed, it doubles the capacity. This is aggressive over-allocation. SF patch <http://www.python.org/sf/578297> has Andrew MacIntyre using PyMalloc in the parser to overcome platform malloc problems in test_longexp on OS/2 EMX. Jack Jansen notes there that it didn't help him on the Mac, because the Mac has problems with frequent ever-growing reallocs, not just with gazillions of teensy mallocs. Win98 has no visible problems with test_longexp, but I tried boosting the test-case size and soon got "senseless" MemoryErrors out of it, and soon after crashed the OS: as I've seen in many other contexts before, while the Win98 realloc remains zippy in bad cases, it leads to extreme fragmentation of user address space, to the point that the OS barfs. I don't yet know whether this fixes Jack's Mac problems, but it does cure Win98's problems when boosting the test case size. It also speeds test_longexp in its unaltered state.
1 parent 059ed83 commit 755ebea

1 file changed

Lines changed: 41 additions & 8 deletions

File tree

‎Parser/node.c‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,25 +18,58 @@ PyNode_New(int type)
1818
return n;
1919
}
2020

21-
#define XXX 3 /* Node alignment factor to speed up realloc */
22-
#define XXXROUNDUP(n) ((n) == 1 ? 1 : ((n) + XXX - 1) / XXX * XXX)
21+
/* See comments at XXXROUNDUP below. */
22+
static int
23+
fancy_roundup(int n)
24+
{
25+
/* Round up to the closest power of 2 >= n. */
26+
int result = 256;
27+
assert(n > 128);
28+
while (result < n)
29+
result <<= 1;
30+
return result;
31+
}
32+
33+
/* A gimmick to make massive numbers of reallocs quicker. The result is
34+
* a number >= the input. For n=0 we must return 0.
35+
* For n=1, we return 1, to avoid wasting memory in common 1-child nodes
36+
* (XXX are those actually common?).
37+
* Else for n <= 128, round up to the closest multiple of 4. Why 4?
38+
* Rounding up to a multiple of an exact power of 2 is very efficient.
39+
* Else call fancy_roundup() to grow proportionately to n. We've got an
40+
* extreme case then (like test_longexp.py), and on many platforms doing
41+
* anything less than proportional growth leads to exorbitant runtime
42+
* (e.g., MacPython), or extreme fragmentation of user address space (e.g.,
43+
* Win98).
44+
* This would be straightforward if a node stored its current capacity. The
45+
* code is tricky to avoid that.
46+
*/
47+
#define XXXROUNDUP(n) ((n) == 1 ? 1 : \
48+
(n) <= 128 ? (((n) + 3) & ~3) : \
49+
fancy_roundup(n))
50+
2351

2452
int
2553
PyNode_AddChild(register node *n1, int type, char *str, int lineno)
2654
{
27-
register int nch = n1->n_nchildren;
28-
register int nch1 = nch+1;
29-
register node *n;
55+
const int nch = n1->n_nchildren;
56+
int current_capacity;
57+
int required_capacity;
58+
node *n;
59+
3060
if (nch == INT_MAX || nch < 0)
3161
return E_OVERFLOW;
32-
if (XXXROUNDUP(nch) < nch1) {
62+
63+
current_capacity = XXXROUNDUP(nch);
64+
required_capacity = XXXROUNDUP(nch + 1);
65+
if (current_capacity < required_capacity) {
3366
n = n1->n_child;
34-
nch1 = XXXROUNDUP(nch1);
35-
PyMem_RESIZE(n, node, nch1);
67+
PyMem_RESIZE(n, node, required_capacity);
3668
if (n == NULL)
3769
return E_NOMEM;
3870
n1->n_child = n;
3971
}
72+
4073
n = &n1->n_child[n1->n_nchildren++];
4174
n->n_type = type;
4275
n->n_str = str;

0 commit comments

Comments
 (0)