1=============================
2How To Validate a New Release
3=============================
4
5.. contents::
6   :local:
7   :depth: 1
8
9Introduction
10============
11
12This document contains information about testing the release candidates that will
13ultimately be the next LLVM release. For more information on how to manage the
14actual release, please refer to :doc:`HowToReleaseLLVM`.
15
16Overview of the Release Process
17-------------------------------
18
19Once the release process starts, the Release Manager will ask for volunteers,
20and it'll be the role of each volunteer to:
21
22* Test and benchmark the previous release
23
24* Test and benchmark each release candidate, comparing to the previous release and candidates
25
26* Identify, reduce and report every regression found during tests and benchmarks
27
28* Make sure the critical bugs get fixed and merged to the next release candidate
29
30Not all bugs or regressions are show-stoppers and it's a bit of a grey area what
31should be fixed before the next candidate and what can wait until the next release.
32
33It'll depend on:
34
35* The severity of the bug, how many people it affects and if it's a regression or a
36  known bug. Known bugs are "unsupported features" and some bugs can be disabled if
37  they have been implemented recently.
38
39* The stage in the release. Less critical bugs should be considered to be fixed between
40  RC1 and RC2, but not so much at the end of it.
41
42* If it's a correctness or a performance regression. Performance regression tends to be
43  taken more lightly than correctness.
44
45.. _scripts:
46
47Scripts
48=======
49
50The scripts are in the ``utils/release`` directory.
51
52test-release.sh
53---------------
54
55This script will check-out, configure and compile LLVM+Clang (+ most add-ons, like ``compiler-rt``,
56``libcxx``, ``libomp`` and ``clang-extra-tools``) in three stages, and will test the final stage.
57It'll have installed the final binaries on the Phase3/Releasei(+Asserts) directory, and
58that's the one you should use for the test-suite and other external tests.
59
60To run the script on a specific release candidate run::
61
62   ./test-release.sh \
63        -release 3.3 \
64        -rc 1 \
65        -no-64bit \
66        -test-asserts \
67        -no-compare-files
68
69Each system will require different options. For instance, x86_64 will obviously not need
70``-no-64bit`` while 32-bit systems will, or the script will fail.
71
72The important flags to get right are:
73
74* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, change it to ``-rc 2`` and so on.
75
76* On non-release testing, you can use ``-final`` in conjunction with ``-no-checkout``, but you'll have to
77  create the ``final`` directory by hand and link the correct source dir to ``final/llvm.src``.
78
79* For release candidates, you need ``-test-asserts``, or it won't create a "Release+Asserts" directory,
80  which is needed for release testing and benchmarking. This will take twice as long.
81
82* On the final candidate you just need Release builds, and that's the binary directory you'll have to pack.
83
84This script builds three phases of Clang+LLVM twice each (Release and Release+Asserts), so use
85screen or nohup to avoid headaches, since it'll take a long time.
86
87Use the ``--help`` option to see all the options and chose it according to your needs.
88
89
90findRegressions-nightly.py
91--------------------------
92
93TODO
94
95.. _test-suite:
96
97Test Suite
98==========
99
100.. contents::
101   :local:
102
103Follow the `LNT Quick Start Guide <http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the test-suite
104
105The binary location you'll have to use for testing is inside the ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``.
106Link that directory to an easier location and run the test-suite.
107
108An example on the run command line, assuming you created a link from the correct
109install directory to ``~/devel/llvm/install``::
110
111   ./sandbox/bin/python sandbox/bin/lnt runtest \
112       nt \
113       -j4 \
114       --sandbox sandbox \
115       --test-suite ~/devel/llvm/test/test-suite \
116       --cc ~/devel/llvm/install/bin/clang \
117       --cxx ~/devel/llvm/install/bin/clang++
118
119It should have no new regressions, compared to the previous release or release candidate. You don't need to fix
120all the bugs in the test-suite, since they're not necessarily meant to pass on all architectures all the time. This is
121due to the nature of the result checking, which relies on direct comparison, and most of the time, the failures are
122related to bad output checking, rather than bad code generation.
123
124If the errors are in LLVM itself, please report every single regression found as blocker, and all the other bugs
125as important, but not necessarily blocking the release to proceed. They can be set as "known failures" and to be
126fix on a future date.
127
128.. _pre-release-process:
129
130Pre-Release Process
131===================
132
133.. contents::
134   :local:
135
136When the release process is announced on the mailing list, you should prepare
137for the testing, by applying the same testing you'll do on the release candidates,
138on the previous release.
139
140You should:
141
142* Download the previous release sources from http://llvm.org/releases/download.html.
143
144* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to ``-final``).
145
146* Once all three stages are done, it'll test the final stage.
147
148* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, run the test-suite.
149
150If the final phase's ``make check-all`` failed, it's a good idea to also test the
151intermediate stages by going on the obj directory and running ``make check-all`` to find
152if there's at least one stage that passes (helps when reducing the error for bug report
153purposes).
154
155.. _release-process:
156
157Release Process
158===============
159
160.. contents::
161   :local:
162
163When the Release Manager sends you the release candidate, download all sources,
164unzip on the same directory (there will be sym-links from the appropriate places
165to them), and run the release test as above.
166
167You should:
168
169* Download the current candidate sources from where the release manager points you
170  (ex. http://llvm.org/pre-releases/3.3/rc1/).
171
172* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the test-suite
173  the same way.
174
175* Compare the results, report all errors on Bugzilla and publish the binary blob
176  where the release manager can grab it.
177
178Once the release manages announces that the latest candidate is the good one, you
179have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` and that
180will be the official binary.
181
182* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory
183
184* Tar that into the same name with ``.tar.gz`` extensioan from outside the directory
185
186* Make it available for the release manager to download
187
188.. _bug-reporting:
189
190Bug Reporting Process
191=====================
192
193.. contents::
194   :local:
195
196If you found regressions or failures when comparing a release candidate with the
197previous release, follow the rules below:
198
199* Critical bugs on compilation should be fixed as soon as possible, possibly before
200  releasing the binary blobs.
201
202* Check-all tests should be fixed before the next release candidate, but can wait
203  until the test-suite run is finished.
204
205* Bugs in the test suite or unimportant check-all tests can be fixed in between
206  release candidates.
207
208* New features or recent big changes, when close to the release, should have done
209  in a way that it's easy to disable. If they misbehave, prefer disabling them than
210  releasing an unstable (but untested) binary package.
211