1==========================
2Source-based Code Coverage
3==========================
4
5.. contents::
6   :local:
7
8Introduction
9============
10
11This document explains how to use clang's source-based code coverage feature.
12It's called "source-based" because it operates on AST and preprocessor
13information directly. This allows it to generate very precise coverage data.
14
15Clang ships two other code coverage implementations:
16
17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
18  various sanitizers. It can provide up to edge-level coverage.
19
20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
21
22From this point onwards "code coverage" will refer to the source-based kind.
23
24The code coverage workflow
25==========================
26
27The code coverage workflow consists of three main steps:
28
29* Compiling with coverage enabled.
30
31* Running the instrumented program.
32
33* Creating coverage reports.
34
35The next few sections work through a complete, copy-'n-paste friendly example
36based on this program:
37
38.. code-block:: cpp
39
40    % cat <<EOF > foo.cc
41    #define BAR(x) ((x) || (x))
42    template <typename T> void foo(T x) {
43      for (unsigned I = 0; I < 10; ++I) { BAR(I); }
44    }
45    int main() {
46      foo<int>(0);
47      foo<float>(0);
48      return 0;
49    }
50    EOF
51
52Compiling with coverage enabled
53===============================
54
55To compile code with coverage enabled, pass ``-fprofile-instr-generate
56-fcoverage-mapping`` to the compiler:
57
58.. code-block:: console
59
60    # Step 1: Compile with coverage enabled.
61    % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
62
63Note that linking together code with and without coverage instrumentation is
64supported: any uninstrumented code simply won't be accounted for.
65
66Running the instrumented program
67================================
68
69The next step is to run the instrumented program. When the program exits it
70will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
71environment variable. If that variable does not exist, the profile is written
72to ``default.profraw`` in the current directory of the program. If
73``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing
74directory structure will be created.  Additionally, the following special
75**pattern strings** are rewritten:
76
77* "%p" expands out to the process ID.
78
79* "%h" expands out to the hostname of the machine running the program.
80
81* "%Nm" expands out to the instrumented binary's signature. When this pattern
82  is specified, the runtime creates a pool of N raw profiles which are used for
83  on-line profile merging. The runtime takes care of selecting a raw profile
84  from the pool, locking it, and updating it before the program exits.  If N is
85  not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must
86  be between 1 and 9. The merge pool specifier can only occur once per filename
87  pattern.
88
89.. code-block:: console
90
91    # Step 2: Run the program.
92    % LLVM_PROFILE_FILE="foo.profraw" ./foo
93
94Creating coverage reports
95=========================
96
97Raw profiles have to be **indexed** before they can be used to generate
98coverage reports. This is done using the "merge" tool in ``llvm-profdata``, so
99named because it can combine and index profiles at the same time:
100
101.. code-block:: console
102
103    # Step 3(a): Index the raw profile.
104    % llvm-profdata merge -sparse foo.profraw -o foo.profdata
105
106There are multiple different ways to render coverage reports. One option is to
107generate a line-oriented report:
108
109.. code-block:: console
110
111    # Step 3(b): Create a line-oriented coverage report.
112    % llvm-cov show ./foo -instr-profile=foo.profdata
113
114To demangle any C++ identifiers in the output, use:
115
116.. code-block:: console
117
118    % llvm-cov show ./foo -instr-profile=foo.profdata | c++filt -n
119
120This report includes a summary view as well as dedicated sub-views for
121templated functions and their instantiations. For our example program, we get
122distinct views for ``foo<int>(...)`` and ``foo<float>(...)``.  If
123``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
124region counts (even in macro expansions):
125
126.. code-block:: none
127
128       20|    1|#define BAR(x) ((x) || (x))
129                               ^20     ^2
130        2|    2|template <typename T> void foo(T x) {
131       22|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
132                                       ^22     ^20  ^20^20
133        2|    4|}
134    ------------------
135    | void foo<int>(int):
136    |      1|    2|template <typename T> void foo(T x) {
137    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
138    |                                     ^11     ^10  ^10^10
139    |      1|    4|}
140    ------------------
141    | void foo<float>(int):
142    |      1|    2|template <typename T> void foo(T x) {
143    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
144    |                                     ^11     ^10  ^10^10
145    |      1|    4|}
146    ------------------
147
148It's possible to generate a file-level summary of coverage statistics (instead
149of a line-oriented report) with:
150
151.. code-block:: console
152
153    # Step 3(c): Create a coverage summary.
154    % llvm-cov report ./foo -instr-profile=foo.profdata
155    Filename                    Regions    Miss   Cover Functions  Executed
156    -----------------------------------------------------------------------
157    /tmp/foo.cc                      13       0 100.00%         3   100.00%
158    -----------------------------------------------------------------------
159    TOTAL                            13       0 100.00%         3   100.00%
160
161A few final notes:
162
163* The ``-sparse`` flag is optional but can result in dramatically smaller
164  indexed profiles. This option should not be used if the indexed profile will
165  be reused for PGO.
166
167* Raw profiles can be discarded after they are indexed. Advanced use of the
168  profile runtime library allows an instrumented program to merge profiling
169  information directly into an existing raw profile on disk. The details are
170  out of scope.
171
172* The ``llvm-profdata`` tool can be used to merge together multiple raw or
173  indexed profiles. To combine profiling data from multiple runs of a program,
174  try e.g:
175
176  .. code-block:: console
177
178      % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
179
180Format compatibility guarantees
181===============================
182
183* There are no backwards or forwards compatibility guarantees for the raw
184  profile format. Raw profiles may be dependent on the specific compiler
185  revision used to generate them. It's inadvisable to store raw profiles for
186  long periods of time.
187
188* Tools must retain **backwards** compatibility with indexed profile formats.
189  These formats are not forwards-compatible: i.e, a tool which uses format
190  version X will not be able to understand format version (X+k).
191
192* There is a third format in play: the format of the coverage mappings emitted
193  into instrumented binaries. Tools must retain **backwards** compatibility
194  with these formats. These formats are not forwards-compatible.
195
196Using the profiling runtime without static initializers
197=======================================================
198
199By default the compiler runtime uses a static initializer to determine the
200profile output path and to register a writer function. To collect profiles
201without using static initializers, do this manually:
202
203* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared
204  library and executable. When the linker finds a definition of this symbol, it
205  knows to skip loading the object which contains the profiling runtime's
206  static initializer.
207
208* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it
209  once from each instrumented executable. This function parses
210  ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files
211  at that path. To get the same behavior without truncating existing files,
212  pass a filename pattern string to ``void __llvm_profile_set_filename(char
213  *)``.  These calls can be placed anywhere so long as they precede all calls
214  to ``__llvm_profile_write_file``.
215
216* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write
217  out a profile. This function returns 0 when it succeeds, and a non-zero value
218  otherwise. Calling this function multiple times appends profile data to an
219  existing on-disk raw profile.
220
221Drawbacks and limitations
222=========================
223
224* Code coverage does not handle unpredictable changes in control flow or stack
225  unwinding in the presence of exceptions precisely. Consider the following
226  function:
227
228  .. code-block:: cpp
229
230      int f() {
231        may_throw();
232        return 0;
233      }
234
235  If the call to ``may_throw()`` propagates an exception into ``f``, the code
236  coverage tool may mark the ``return`` statement as executed even though it is
237  not. A call to ``longjmp()`` can have similar effects.
238