|
| 1 | +.. _buildbots: |
| 2 | + |
| 3 | +Continuous Integration |
| 4 | +====================== |
| 5 | + |
| 6 | +To assert that there are no regressions in the :doc:`development and maintenance |
| 7 | +branches <devcycle>`, Python has a set of dedicated machines (called *buildbots* or |
| 8 | +*build slaves*) used for continuous integration. They span a number of |
| 9 | +hardware/operating system combinations. Furthermore, each machine hosts |
| 10 | +several *builders*, one per active branch: when a new change is pushed |
| 11 | +to this branch on the public Mercurial repository, all corresponding builders |
| 12 | +will schedule a new build to be run as soon as possible. |
| 13 | + |
| 14 | +The build steps run by the buildbots are the following: |
| 15 | + |
| 16 | +* Checkout of the source tree for the changeset which triggered the build |
| 17 | +* Compiling Python |
| 18 | +* Running the test suite using :ref:`strenuous settings <strenuous_testing>` |
| 19 | +* Cleaning up the build tree |
| 20 | + |
| 21 | +It is your responsibility, as a core developer, to check the automatic |
| 22 | +build results after you push a change to the repository. It is therefore |
| 23 | +important that you get acquainted with the way these results are presented, |
| 24 | +and how various kinds of failures can be explained and diagnosed. |
| 25 | + |
| 26 | +Checking results of automatic builds |
| 27 | +------------------------------------ |
| 28 | + |
| 29 | +There are two ways of visualizing recent build results: |
| 30 | + |
| 31 | +* The Web interface for each branch at http://python.org/dev/buildbot/, |
| 32 | + where the so-called "waterfall" view presents a vertical rundown of recent |
| 33 | + builds for each builder. When interested in one build, you'll have to |
| 34 | + click on it to know which changesets it corresponds to. |
| 35 | + |
| 36 | +* The command-line ``bbreport.py`` client, which you can get from |
| 37 | + http://code.google.com/p/bbreport/. Installing it is trivial: just add |
| 38 | + the directory containing ``bbreport.py`` to your system path so that |
| 39 | + you can run it from any filesystem location. For example, if you want |
| 40 | + to display the latest build results on the development ("default") branch, |
| 41 | + type:: |
| 42 | + |
| 43 | + bbreport.py -q 3.x |
| 44 | + |
| 45 | +Some buildbots are much faster than others. Over time, you will learn which |
| 46 | +ones produce the quickest results after a build, and which ones take the |
| 47 | +longest time. |
| 48 | + |
| 49 | +Also, when several changesets are pushed in a quick succession in the same |
| 50 | +branch, it often happens that a single build is scheduled for all these |
| 51 | +changesets. |
| 52 | + |
| 53 | +Stability |
| 54 | +--------- |
| 55 | + |
| 56 | +A subset of the buildbots are marked "stable". They are taken into account |
| 57 | +when making a new release. The rule is that no stable buildbot must witness |
| 58 | +persistent failures when the release is cut. It is absolutely **vital** |
| 59 | +that core developers fix any issue they introduce on the stable buildbots, |
| 60 | +as soon as possible. |
| 61 | + |
| 62 | +This does not mean that other builders' test results can be taken lightly, |
| 63 | +either. Some of them are known for having platform-specific issues that |
| 64 | +prevent some tests from succeeding (or even terminating at all), but |
| 65 | +introducing additional failures should generally not be an option. |
| 66 | + |
| 67 | +Flags-dependent failures |
| 68 | +------------------------ |
| 69 | + |
| 70 | +Sometimes, while you have run the :doc:`whole test suite <runtests>` before |
| 71 | +committing, you may witness unexpected failures on the buildbots. One source |
| 72 | +of such discrepancies is if different flags have been passed to the test runner |
| 73 | +or to Python itself. To reproduce, make sure you use the same flags as the |
| 74 | +buildbots: they can be found out simply by clicking the **stdio** link for |
| 75 | +the failing build's tests. For example:: |
| 76 | + |
| 77 | + ./python.exe -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW |
| 78 | + |
| 79 | +.. note:: |
| 80 | + Running ``Lib/test/regrtest.py`` is exactly equivalent to running |
| 81 | + ``-m test``. |
| 82 | + |
| 83 | +Ordering-dependent failures |
| 84 | +--------------------------- |
| 85 | + |
| 86 | +Sometimes even the failure is subtler, as it relies on the order in which |
| 87 | +the tests are run. The buildbots *randomize* test order (by using the ``-r`` |
| 88 | +option to the test runner) to maximize the probability that potential |
| 89 | +interferences between library modules are exercised; the downside is that it |
| 90 | +can make for seemingly sporadic failures. |
| 91 | + |
| 92 | +The ``--randseed`` option makes it easy to reproduce the exact randomization |
| 93 | +used in a given build. Again, open the ``stdio`` link for the failing test |
| 94 | +run, and check the beginning of the test output proper. |
| 95 | + |
| 96 | +Let's assume, for the sake of example, that the output starts with:: |
| 97 | + |
| 98 | + ./python -Wd -E -bb Lib/test/regrtest.py -uall -rwW |
| 99 | + == CPython 3.3a0 (default:22ae2b002865, Mar 30 2011, 13:58:40) [GCC 4.4.5] |
| 100 | + == Linux-2.6.36-gentoo-r5-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-1.12.14 little-endian |
| 101 | + == /home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/build/test_python_29628 |
| 102 | + Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0) |
| 103 | + Using random seed 2613169 |
| 104 | + [ 1/353] test_augassign |
| 105 | + [ 2/353] test_functools |
| 106 | + |
| 107 | +You can reproduce the exact same order using:: |
| 108 | + |
| 109 | + ./python -Wd -E -bb -m test -uall -rwW --randseed 2613169 |
| 110 | + |
| 111 | +It will run the following sequence (trimmed for brevity):: |
| 112 | + |
| 113 | + [ 1/353] test_augassign |
| 114 | + [ 2/353] test_functools |
| 115 | + [ 3/353] test_bool |
| 116 | + [ 4/353] test_contains |
| 117 | + [ 5/353] test_compileall |
| 118 | + [ 6/353] test_unicode |
| 119 | + |
| 120 | +If this is enough to reproduce the failure on your setup, you can then |
| 121 | +bisect the test sequence to look for the specific interference causing the |
| 122 | +failure. Copy and paste the test sequence in a text file, then use the |
| 123 | +``--fromfile`` (or ``-f``) option of the test runner to run the exact |
| 124 | +sequence recorded in that text file:: |
| 125 | + |
| 126 | + ./python -Wd -E -bb -m test -uall -rwW --fromfile mytestsequence.txt |
| 127 | + |
| 128 | +In the example sequence above, if ``test_unicode`` had failed, you would |
| 129 | +first test the following sequence:: |
| 130 | + |
| 131 | + [ 1/353] test_augassign |
| 132 | + [ 2/353] test_functools |
| 133 | + [ 3/353] test_bool |
| 134 | + [ 6/353] test_unicode |
| 135 | + |
| 136 | +And, if it succeeds, the following one instead (which, hopefully, shall |
| 137 | +fail):: |
| 138 | + |
| 139 | + [ 4/353] test_contains |
| 140 | + [ 5/353] test_compileall |
| 141 | + [ 6/353] test_unicode |
| 142 | + |
| 143 | +Then, recursively, narrow down the search until you get a single pair of |
| 144 | +tests which triggers the failure. It is very rare that such an interference |
| 145 | +involves more than **two** tests. If this is the case, we can only wish you |
| 146 | +good luck! |
| 147 | + |
| 148 | +.. note:: |
| 149 | + You cannot use the ``-j`` option (for parallel testing) when diagnosing |
| 150 | + ordering-dependent failures. Using ``-j`` isolates each test in a |
| 151 | + pristine subprocess and, therefore, prevents you from reproducing any |
| 152 | + interference between tests. |
| 153 | + |
| 154 | + |
| 155 | +Transient failures |
| 156 | +------------------ |
| 157 | + |
| 158 | +While we try to make the test suite as reliable as possible, some tests do |
| 159 | +not reach a perfect level of reproduceability. Some of them will sometimes |
| 160 | +display spurious failures, depending on various conditions. Here are common |
| 161 | +offenders: |
| 162 | + |
| 163 | +* Network-related tests, such as ``test_poplib``, ``test_urllibnet``, etc. |
| 164 | + Their failures can stem from adverse network conditions, or imperfect |
| 165 | + thread synchronization in the test code, which often has to run a |
| 166 | + server in a separate thread. |
| 167 | + |
| 168 | +* Tests dealing with delicate issues such as inter-thread or inter-process |
| 169 | + synchronization, or Unix signals: ``test_multiprocessing``, |
| 170 | + ``test_threading``, ``test_subprocess``, ``test_threadsignals``. |
| 171 | + |
| 172 | +When you think a failure might be transient, it is recommended you confirm by |
| 173 | +waiting for the next build. Still, even if the failure does turn out sporadic |
| 174 | +and unpredictable, the issue should be reported on the bug tracker; even |
| 175 | +better if it can be diagnosed and suppressed by fixing the test's implementation, |
| 176 | +or by making its parameters - such as a timeout - more robust. |
| 177 | + |
0 commit comments