PyPerformance Results on Windows for Py310 --> Py311 #315

sweeneyde · 2022-03-08T02:12:42Z

sweeneyde
Mar 8, 2022

Comparing versions (installed from python.org installers):

Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Python 3.11.0a6 (main, Mar  7 2022, 16:46:19) [MSC v.1929 64 bit (AMD64)] on win32

py -m pyperf compare_to .\310.json .\311.json -G
Slower (20):
- unpack_sequence: 109 ns +- 1 ns -> 178 ns +- 2 ns: 1.63x slower
- nbody: 161 ms +- 1 ms -> 187 ms +- 2 ms: 1.17x slower
- scimark_fft: 409 ms +- 5 ms -> 471 ms +- 7 ms: 1.15x slower
- pathlib: 187 ms +- 2 ms -> 211 ms +- 2 ms: 1.13x slower
- nqueens: 111 ms +- 1 ms -> 123 ms +- 9 ms: 1.10x slower
- fannkuch: 505 ms +- 3 ms -> 553 ms +- 1 ms: 1.10x slower
- unpickle_pure_python: 321 us +- 4 us -> 351 us +- 6 us: 1.09x slower
- meteor_contest: 119 ms +- 2 ms -> 128 ms +- 1 ms: 1.07x slower
- scimark_sparse_mat_mult: 5.59 ms +- 0.61 ms -> 5.94 ms +- 0.37 ms: 1.06x slower
- xml_etree_iterparse: 112 ms +- 2 ms -> 117 ms +- 2 ms: 1.05x slower
- xml_etree_generate: 106 ms +- 1 ms -> 110 ms +- 1 ms: 1.04x slower
- pidigits: 211 ms +- 1 ms -> 219 ms +- 2 ms: 1.04x slower
- sympy_expand: 576 ms +- 4 ms -> 592 ms +- 2 ms: 1.03x slower
- regex_v8: 28.2 ms +- 0.3 ms -> 28.8 ms +- 0.4 ms: 1.02x slower
- telco: 6.98 ms +- 0.08 ms -> 7.12 ms +- 0.06 ms: 1.02x slower
- sympy_str: 355 ms +- 3 ms -> 361 ms +- 8 ms: 1.02x slower
- regex_effbot: 3.38 ms +- 0.03 ms -> 3.43 ms +- 0.06 ms: 1.01x slower
- spectral_norm: 157 ms +- 3 ms -> 159 ms +- 1 ms: 1.01x slower
- json_loads: 27.5 us +- 0.2 us -> 27.8 us +- 0.2 us: 1.01x slower
- xml_etree_parse: 161 ms +- 2 ms -> 162 ms +- 3 ms: 1.01x slower

Faster (35):
- deltablue: 7.62 ms +- 0.17 ms -> 5.59 ms +- 0.06 ms: 1.36x faster
- go: 236 ms +- 2 ms -> 191 ms +- 1 ms: 1.23x faster
- logging_silent: 185 ns +- 6 ns -> 151 ns +- 4 ns: 1.23x faster
- pyflate: 721 ms +- 33 ms -> 616 ms +- 4 ms: 1.17x faster
- xml_etree_process: 93.2 ms +- 36.5 ms -> 79.7 ms +- 0.8 ms: 1.17x faster
- richards: 74.7 ms +- 1.1 ms -> 63.9 ms +- 1.0 ms: 1.17x faster
- scimark_sor: 210 ms +- 3 ms -> 182 ms +- 1 ms: 1.15x faster
- raytrace: 501 ms +- 6 ms -> 441 ms +- 13 ms: 1.14x faster
- python_startup: 34.4 ms +- 0.2 ms -> 30.4 ms +- 0.1 ms: 1.13x faster
- 2to3: 439 ms +- 23 ms -> 391 ms +- 3 ms: 1.12x faster
- tornado_http: 216 ms +- 3 ms -> 193 ms +- 3 ms: 1.12x faster
- scimark_lu: 173 ms +- 3 ms -> 155 ms +- 2 ms: 1.12x faster
- html5lib: 88.1 ms +- 3.8 ms -> 79.0 ms +- 2.8 ms: 1.12x faster
- scimark_monte_carlo: 108 ms +- 2 ms -> 97.5 ms +- 0.6 ms: 1.11x faster
- logging_format: 13.8 us +- 0.2 us -> 12.6 us +- 0.2 us: 1.09x faster
- mako: 16.9 ms +- 0.2 ms -> 15.6 ms +- 0.7 ms: 1.08x faster
- crypto_pyaes: 118 ms +- 1 ms -> 110 ms +- 1 ms: 1.07x faster
- unpickle: 15.9 us +- 1.0 us -> 14.9 us +- 0.2 us: 1.07x faster
- logging_simple: 12.4 us +- 0.2 us -> 11.6 us +- 0.5 us: 1.07x faster
- sqlite_synth: 3.39 us +- 0.03 us -> 3.20 us +- 0.03 us: 1.06x faster
- pickle_pure_python: 470 us +- 5 us -> 447 us +- 6 us: 1.05x faster
- unpickle_list: 5.41 us +- 0.06 us -> 5.15 us +- 0.05 us: 1.05x faster
- float: 128 ms +- 1 ms -> 123 ms +- 3 ms: 1.04x faster
- pickle_list: 5.52 us +- 0.05 us -> 5.30 us +- 0.05 us: 1.04x faster
- pickle_dict: 36.5 us +- 0.2 us -> 35.1 us +- 0.3 us: 1.04x faster
- python_startup_no_site: 26.3 ms +- 0.2 ms -> 25.4 ms +- 0.1 ms: 1.04x faster
- json_dumps: 14.7 ms +- 0.2 ms -> 14.2 ms +- 0.3 ms: 1.03x faster
- regex_compile: 186 ms +- 1 ms -> 182 ms +- 3 ms: 1.03x faster
- sympy_sum: 209 ms +- 4 ms -> 203 ms +- 6 ms: 1.03x faster
- hexiom: 10.1 ms +- 0.2 ms -> 9.93 ms +- 0.07 ms: 1.02x faster
- pickle: 13.8 us +- 0.1 us -> 13.5 us +- 0.1 us: 1.02x faster
- chameleon: 10.6 ms +- 0.2 ms -> 10.4 ms +- 0.2 ms: 1.01x faster
- django_template: 52.4 ms +- 0.9 ms -> 51.8 ms +- 0.9 ms: 1.01x faster
- sympy_integrate: 25.8 ms +- 0.4 ms -> 25.6 ms +- 0.4 ms: 1.01x faster
- regex_dna: 212 ms +- 1 ms -> 211 ms +- 3 ms: 1.01x faster

Benchmark hidden because not significant (1): chaos

Geometric mean: 1.03x faster
Ignored benchmarks (2) of .\310.json: genshi_text, genshi_xml

I replicated the nbody results on another run of each benchmark:

Performance version: 1.0.4
Python version: 3.10.2 (64-bit) revision a58ebcc
Report on Windows-10-10.0.19044-SP0
Number of logical CPUs: 12
Start date: 2022-03-07 21:07:25.016445
End date: 2022-03-07 21:07:41.532414

### nbody ###
Mean +- std dev: 161 ms +- 2 ms

Performance version: 1.0.4
Python version: 3.11.0a6 (64-bit)
Report on Windows-10-10.0.19044-SP0
Number of logical CPUs: 12
Start date: 2022-03-07 21:06:44.511546
End date: 2022-03-07 21:07:03.186641

### nbody ###
Mean +- std dev: 189 ms +- 7 ms

I did have to change a couple of lines of pyperformance to get it to run:

In _pythoninfo.py, change base_executable = os.path.join(os.path.dirname(stdlib_dir), 'python') to python.exe.

sweeneyde · 2022-03-08T03:45:59Z

sweeneyde
Mar 8, 2022
Author

Apparently it's the PGO that is hurting nbody more than it is helping :(

Results on main right now for .\PCBuild\build.bat versus .\PCBuild\build.bat --pgo:

Mean +- std dev: [main_nbody_release] 160 ms +- 2 ms -> [main_nbody_pgo] 183 ms +- 2 ms: 1.14x slower

Could be related to https://bugs.python.org/issue45116

0 replies

gvanrossum · 2022-03-08T04:08:03Z

gvanrossum
Mar 8, 2022
Maintainer

This could also be bpo-45116. Is there an experiment that you could do to force inlining in ceval.c?

3% faster is really disappointing.

0 replies

brandtbucher · 2022-03-08T05:29:24Z

brandtbucher
Mar 8, 2022
Maintainer

We should also recalculate that geometric mean without the unpack_sequence result. That slowdown is because inline caching pushed the size of that function slightly beyond our old quickening threshold. It was fixed on main earlier today.

1 reply

sweeneyde Mar 8, 2022
Author

py -m pyperf compare_to .\310.json .\311.json --benchmark nbody --benchmark scimark_fft --benchmark pathlib --benchmark nqueens --benchmark fannkuch --benchmark unpickle_pure_python --benchmark meteor_contest --benchmark scimark_sparse_mat_mult --benchmark xml_etree_iterparse --benchmark xml_etree_generate --benchmark pidigits --benchmark sympy_expand --benchmark regex_v8 --benchmark telco --benchmark sympy_str --benchmark regex_effbot --benchmark spectral_norm --benchmark json_loads --benchmark xml_etree_parse --benchmark deltablue --benchmark go --benchmark logging_silent --benchmark pyflate --benchmark xml_etree_process --benchmark richards --benchmark scimark_sor --benchmark raytrace --benchmark python_startup --benchmark 2to3 --benchmark tornado_http --benchmark scimark_lu --benchmark html5lib --benchmark scimark_monte_carlo --benchmark logging_format --benchmark mako --benchmark crypto_pyaes --benchmark unpickle --benchmark logging_simple --benchmark sqlite_synth --benchmark pickle_pure_python --benchmark unpickle_list --benchmark float --benchmark pickle_list --benchmark pickle_dict --benchmark python_startup_no_site --benchmark json_dumps --benchmark regex_compile --benchmark sympy_sum --benchmark hexiom --benchmark pickle --benchmark chameleon --benchmark django_template --benchmark sympy_integrate --benchmark regex_dna -G

(all the same stuff, except unpack_sequence was printed here)

Geometric mean: 1.04x faster

kumaraditya303 · 2022-03-08T05:46:10Z

kumaraditya303
Mar 8, 2022

Perhaps instead of msvc, clang-cl can be used instead as it seems supported.

https://docs.microsoft.com/en-us/cpp/build/clang-support-msbuild?view=msvc-170#configure-a-windows-project-to-use-clang-tools

0 replies

gvanrossum · 2022-03-08T16:24:41Z

gvanrossum
Mar 8, 2022
Maintainer

@kumaraditya303 If you're interested, it's worth at least trying the Clang stuff, to see just how much we'd have to change to get a running binary with PGO+LTO that can still load 3rd party extensions (DLLs) built using the standard MSVC toolchain (e.g. wheels).

1 reply

kumaraditya303 Mar 9, 2022

There is already a PR for it but it has conflicts python/cpython#18371

gvanrossum · 2022-03-09T16:55:11Z

gvanrossum
Mar 9, 2022
Maintainer

Do you want to use that as a starting point? IMO most of the work will be experimenting with how easy it is to use (e.g. make it easy for a dinosaur like myself to switch).

0 replies

gvanrossum · 2022-03-15T21:25:57Z

gvanrossum
Mar 15, 2022
Maintainer

I am still struggling to repro this result, mostly because pyperformance has a few issues.

@sweeneyde you nailed one (base_executable is wrong for Windows, although the right fix is more complex).
Another I ran into is that psutil is required to do something to the process priority, and it's not trivial to get this installed in the sub-venvs where it is needed. @sweeneyde Did you run into this? Did you manage to fix it?
A third is Creating venvs takes very long on Windows python/pyperformance#146, there's too much venv overhead.

Eric is working on the latter two now.

In the meantime I am trying to think of a fix. Mark suggested that we might redefine Py_INCREF/DECREF as macros at the top of the file to force the compiler to "inline" them. Are there other suggestions/ideas?

1 reply

sweeneyde Mar 15, 2022
Author

Another I ran into is that psutil is required to do something to the process priority, and it's not trivial to get this installed in the sub-venvs where it is needed. @sweeneyde Did you run into this? Did you manage to fix it?

I got the process priority warnings but ignored them. I did not do anything to change process priority (I never tried too hard to get pyperf system tune to work), but I did turn my laptop to airplane mode (after everything was installed, while the first benchmark was running) and then avoided doing much with it while running those benchmarks.

gvanrossum · 2022-03-16T23:47:32Z

gvanrossum
Mar 16, 2022
Maintainer

I get a different set of results. The geometric mean is 1.13x faster. (But I have 14 failing benchmarks.) This is on yesterday's main branch (sadly I failed to get the exact commit). A few random comments:

nbody: 1.16x slower, comparable to Dennis' result
unpack_sequence: 1.23x faster, whereas Dennis had it 1.63x slower
The other four that are slower in my run are similarly slower in Dennis' run
python_startup and and python_startup_no_site are much faster in my run (maybe deepfreeze works better now?)
raytrace is much faster, as is unpickle (why?)
deltablue is only a little bit faster (1.40x vs. 1.36x)

I gave up at this point, but you get the picture -- I believe my results aren't completely off. Possibly the age or architecture of the hardware differs?

Details

Slower (5):
- nbody: 138 ms +- 9 ms -> 160 ms +- 7 ms: 1.16x slower
- scimark_fft: 398 ms +- 28 ms -> 449 ms +- 21 ms: 1.13x slower
- scimark_sparse_mat_mult: 5.59 ms +- 0.57 ms -> 6.21 ms +- 0.71 ms: 1.11x slower
- meteor_contest: 123 ms +- 7 ms -> 134 ms +- 8 ms: 1.09x slower
- fannkuch: 497 ms +- 15 ms -> 541 ms +- 13 ms: 1.09x slower

Faster (33):
- python_startup_no_site: 45.5 ms +- 9.6 ms -> 28.2 ms +- 6.9 ms: 1.61x faster
- python_startup: 51.4 ms +- 7.0 ms -> 33.7 ms +- 9.3 ms: 1.52x faster
- raytrace: 641 ms +- 74 ms -> 441 ms +- 18 ms: 1.45x faster
- unpickle: 20.6 us +- 5.2 us -> 14.5 us +- 0.9 us: 1.42x faster
- deltablue: 7.84 ms +- 0.63 ms -> 5.62 ms +- 0.25 ms: 1.40x faster
- richards: 88.8 ms +- 7.1 ms -> 64.6 ms +- 3.5 ms: 1.38x faster
- scimark_sor: 240 ms +- 28 ms -> 187 ms +- 18 ms: 1.29x faster
- regex_v8: 34.4 ms +- 4.2 ms -> 27.6 ms +- 1.6 ms: 1.25x faster
- regex_effbot: 3.77 ms +- 0.40 ms -> 3.03 ms +- 0.16 ms: 1.24x faster
- unpickle_list: 6.36 us +- 1.42 us -> 5.14 us +- 0.50 us: 1.24x faster
- unpack_sequence: 99.1 ns +- 8.9 ns -> 80.6 ns +- 3.9 ns: 1.23x faster
- scimark_monte_carlo: 125 ms +- 12 ms -> 102 ms +- 6 ms: 1.23x faster
- go: 249 ms +- 16 ms -> 203 ms +- 13 ms: 1.23x faster
- chaos: 124 ms +- 10 ms -> 102 ms +- 5 ms: 1.22x faster
- sqlite_synth: 3.72 us +- 0.32 us -> 3.08 us +- 0.21 us: 1.21x faster
- regex_dna: 252 ms +- 22 ms -> 211 ms +- 10 ms: 1.19x faster
- 2to3: 530 ms +- 42 ms -> 445 ms +- 36 ms: 1.19x faster
- logging_silent: 181 ns +- 10 ns -> 153 ns +- 20 ns: 1.18x faster
- scimark_lu: 197 ms +- 17 ms -> 167 ms +- 11 ms: 1.18x faster
- logging_simple: 11.6 us +- 0.7 us -> 9.91 us +- 0.47 us: 1.17x faster
- pyflate: 749 ms +- 26 ms -> 642 ms +- 22 ms: 1.17x faster
- pickle_pure_python: 521 us +- 109 us -> 447 us +- 31 us: 1.17x faster
- float: 125 ms +- 10 ms -> 112 ms +- 5 ms: 1.12x faster
- xml_etree_process: 96.8 ms +- 5.9 ms -> 88.0 ms +- 6.1 ms: 1.10x faster
- xml_etree_parse: 197 ms +- 16 ms -> 182 ms +- 11 ms: 1.08x faster
- json_loads: 27.1 us +- 1.3 us -> 25.2 us +- 2.1 us: 1.08x faster
- pickle: 13.3 us +- 0.9 us -> 12.4 us +- 0.6 us: 1.07x faster
- spectral_norm: 169 ms +- 14 ms -> 159 ms +- 9 ms: 1.06x faster
- xml_etree_iterparse: 140 ms +- 15 ms -> 132 ms +- 8 ms: 1.06x faster
- telco: 7.98 ms +- 0.74 ms -> 7.62 ms +- 0.39 ms: 1.05x faster
- json_dumps: 14.7 ms +- 0.8 ms -> 14.1 ms +- 0.7 ms: 1.04x faster
- pidigits: 269 ms +- 16 ms -> 259 ms +- 11 ms: 1.04x faster
- pickle_dict: 32.9 us +- 1.7 us -> 32.1 us +- 1.8 us: 1.03x faster

Benchmark hidden because not significant (7): hexiom, logging_format, nqueens, pathlib, pickle_list, unpickle_pure_python, xml_etree_generate

Geometric mean: 1.13x faster
Ignored benchmarks (14) of .\310.json: chameleon, crypto_pyaes, django_template, genshi_text, genshi_xml, html5lib, mako, sqlalchemy_declarative, sqlalchemy_imperative, sympy_expand, sympy_integrate, sympy_str, sympy_sum, tornado_http

I would attach my JSON files but GitHub doesn't believe in JSON attachments.

0 replies

gvanrossum · 2022-03-16T23:53:23Z

gvanrossum
Mar 16, 2022
Maintainer

More details: the log of all changes to ceval.c since 3.11a6.

Details

C:> g log v3.11.0a6..main .\Python\ceval.c 
commit 49e1e1e1bd59cac1855b1ef4dec05d649ebcd81a
Author: Mark Shannon <mark@hotpy.org>
Date:   Wed Mar 16 00:08:37 2022 +0000

    bpo-46841: Don't scan backwards in bytecode (GH-31901)

commit 099f75614100e88ed90b68d20a51a8d9c22f81a7
Author: Mark Shannon <mark@hotpy.org>
Date:   Tue Mar 15 17:06:21 2022 +0000

    bpo-45923: Decouple suspension of tracing from tracing flag. (GH-31908)

commit 13b041222399152acb555337572bd1d571734984
Author: Oleg Iarygin <oleg@arhadthedev.net>
Date:   Mon Mar 14 19:04:22 2022 +0300

    bpo-46920: Remove code that has explainers why it was disabled (GH-31813)

commit a89c29fbcc7e7e85848499443d819c3fab68c78a
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Fri Mar 11 16:31:50 2022 -0800

    bpo-46841: Add a _Py_SET_OPCODE macro (GH-31780)

commit 5498a61c7c25db6f9e76032aa9c5153d79e09889
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Tue Mar 8 07:53:22 2022 -0800

    bpo-46841: Don't use an oparg counter for `STORE_SUBSCR` (GH-31742)

commit f193631387bfee99a812e39b05d5b7e6384b57f5
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Mon Mar 7 11:45:00 2022 -0800

    bpo-46841: Use inline caching for calls (GH-31709)

0 replies

gvanrossum · 2022-03-17T05:02:26Z

gvanrossum
Mar 17, 2022
Maintainer

Another build (a day later) is 5% faster again, but I don't really believe these results. But that means I also don't really believe Dennis' results. We're waiting on a stable machine to run tests.

1 reply

sweeneyde Mar 17, 2022
Author

I'd be more than happy to be proven wrong! I can try to re-run things within the next couple of days as well.

PyPerformance Results on Windows for Py310 --> Py311 #315

Uh oh!

Uh oh!

sweeneyde Mar 8, 2022

Replies: 10 comments · 4 replies

Uh oh!

Uh oh!

sweeneyde Mar 8, 2022 Author

Uh oh!

Uh oh!

gvanrossum Mar 8, 2022 Maintainer

Uh oh!

brandtbucher Mar 8, 2022 Maintainer

Uh oh!

sweeneyde Mar 8, 2022 Author

Uh oh!

kumaraditya303 Mar 8, 2022

Uh oh!

gvanrossum Mar 8, 2022 Maintainer

Uh oh!

kumaraditya303 Mar 9, 2022

Uh oh!

gvanrossum Mar 9, 2022 Maintainer

Uh oh!

gvanrossum Mar 15, 2022 Maintainer

Uh oh!

Uh oh!

sweeneyde Mar 15, 2022 Author

Uh oh!

gvanrossum Mar 16, 2022 Maintainer

Uh oh!

gvanrossum Mar 16, 2022 Maintainer

Uh oh!

gvanrossum Mar 17, 2022 Maintainer

Uh oh!

sweeneyde Mar 17, 2022 Author

sweeneyde
Mar 8, 2022

Replies: 10 comments 4 replies

sweeneyde
Mar 8, 2022
Author

gvanrossum
Mar 8, 2022
Maintainer

brandtbucher
Mar 8, 2022
Maintainer

sweeneyde Mar 8, 2022
Author

kumaraditya303
Mar 8, 2022

gvanrossum
Mar 8, 2022
Maintainer

gvanrossum
Mar 9, 2022
Maintainer

gvanrossum
Mar 15, 2022
Maintainer

sweeneyde Mar 15, 2022
Author

gvanrossum
Mar 16, 2022
Maintainer

gvanrossum
Mar 16, 2022
Maintainer

gvanrossum
Mar 17, 2022
Maintainer

sweeneyde Mar 17, 2022
Author