1====================================
2LLVM bugpoint tool: design and usage
3====================================
4
5.. contents::
6   :local:
7
8Description
9===========
10
11``bugpoint`` narrows down the source of problems in LLVM tools and passes.  It
12can be used to debug three types of failures: optimizer crashes, miscompilations
13by optimizers, or bad native code generation (including problems in the static
14and JIT compilers).  It aims to reduce large test cases to small, useful ones.
15For example, if ``opt`` crashes while optimizing a file, it will identify the
16optimization (or combination of optimizations) that causes the crash, and reduce
17the file down to a small example which triggers the crash.
18
19For detailed case scenarios, such as debugging ``opt``, or one of the LLVM code
20generators, see :doc:`HowToSubmitABug`.
21
22Design Philosophy
23=================
24
25``bugpoint`` is designed to be a useful tool without requiring any hooks into
26the LLVM infrastructure at all.  It works with any and all LLVM passes and code
27generators, and does not need to "know" how they work.  Because of this, it may
28appear to do stupid things or miss obvious simplifications.  ``bugpoint`` is
29also designed to trade off programmer time for computer time in the
30compiler-debugging process; consequently, it may take a long period of
31(unattended) time to reduce a test case, but we feel it is still worth it. Note
32that ``bugpoint`` is generally very quick unless debugging a miscompilation
33where each test of the program (which requires executing it) takes a long time.
34
35Automatic Debugger Selection
36----------------------------
37
38``bugpoint`` reads each ``.bc`` or ``.ll`` file specified on the command line
39and links them together into a single module, called the test program.  If any
40LLVM passes are specified on the command line, it runs these passes on the test
41program.  If any of the passes crash, or if they produce malformed output (which
42causes the verifier to abort), ``bugpoint`` starts the `crash debugger`_.
43
44Otherwise, if the ``-output`` option was not specified, ``bugpoint`` runs the
45test program with the "safe" backend (which is assumed to generate good code) to
46generate a reference output.  Once ``bugpoint`` has a reference output for the
47test program, it tries executing it with the selected code generator.  If the
48selected code generator crashes, ``bugpoint`` starts the `crash debugger`_ on
49the code generator.  Otherwise, if the resulting output differs from the
50reference output, it assumes the difference resulted from a code generator
51failure, and starts the `code generator debugger`_.
52
53Finally, if the output of the selected code generator matches the reference
54output, ``bugpoint`` runs the test program after all of the LLVM passes have
55been applied to it.  If its output differs from the reference output, it assumes
56the difference resulted from a failure in one of the LLVM passes, and enters the
57`miscompilation debugger`_.  Otherwise, there is no problem ``bugpoint`` can
58debug.
59
60.. _crash debugger:
61
62Crash debugger
63--------------
64
65If an optimizer or code generator crashes, ``bugpoint`` will try as hard as it
66can to reduce the list of passes (for optimizer crashes) and the size of the
67test program.  First, ``bugpoint`` figures out which combination of optimizer
68passes triggers the bug. This is useful when debugging a problem exposed by
69``opt``, for example, because it runs over 38 passes.
70
71Next, ``bugpoint`` tries removing functions from the test program, to reduce its
72size.  Usually it is able to reduce a test program to a single function, when
73debugging intraprocedural optimizations.  Once the number of functions has been
74reduced, it attempts to delete various edges in the control flow graph, to
75reduce the size of the function as much as possible.  Finally, ``bugpoint``
76deletes any individual LLVM instructions whose absence does not eliminate the
77failure.  At the end, ``bugpoint`` should tell you what passes crash, give you a
78bitcode file, and give you instructions on how to reproduce the failure with
79``opt`` or ``llc``.
80
81.. _code generator debugger:
82
83Code generator debugger
84-----------------------
85
86The code generator debugger attempts to narrow down the amount of code that is
87being miscompiled by the selected code generator.  To do this, it takes the test
88program and partitions it into two pieces: one piece which it compiles with the
89"safe" backend (into a shared object), and one piece which it runs with either
90the JIT or the static LLC compiler.  It uses several techniques to reduce the
91amount of code pushed through the LLVM code generator, to reduce the potential
92scope of the problem.  After it is finished, it emits two bitcode files (called
93"test" [to be compiled with the code generator] and "safe" [to be compiled with
94the "safe" backend], respectively), and instructions for reproducing the
95problem.  The code generator debugger assumes that the "safe" backend produces
96good code.
97
98.. _miscompilation debugger:
99
100Miscompilation debugger
101-----------------------
102
103The miscompilation debugger works similarly to the code generator debugger.  It
104works by splitting the test program into two pieces, running the optimizations
105specified on one piece, linking the two pieces back together, and then executing
106the result.  It attempts to narrow down the list of passes to the one (or few)
107which are causing the miscompilation, then reduce the portion of the test
108program which is being miscompiled.  The miscompilation debugger assumes that
109the selected code generator is working properly.
110
111Advice for using bugpoint
112=========================
113
114``bugpoint`` can be a remarkably useful tool, but it sometimes works in
115non-obvious ways.  Here are some hints and tips:
116
117* In the code generator and miscompilation debuggers, ``bugpoint`` only works
118  with programs that have deterministic output.  Thus, if the program outputs
119  ``argv[0]``, the date, time, or any other "random" data, ``bugpoint`` may
120  misinterpret differences in these data, when output, as the result of a
121  miscompilation.  Programs should be temporarily modified to disable outputs
122  that are likely to vary from run to run.
123
124* In the code generator and miscompilation debuggers, debugging will go faster
125  if you manually modify the program or its inputs to reduce the runtime, but
126  still exhibit the problem.
127
128* ``bugpoint`` is extremely useful when working on a new optimization: it helps
129  track down regressions quickly.  To avoid having to relink ``bugpoint`` every
130  time you change your optimization however, have ``bugpoint`` dynamically load
131  your optimization with the ``-load`` option.
132
133* ``bugpoint`` can generate a lot of output and run for a long period of time.
134  It is often useful to capture the output of the program to file.  For example,
135  in the C shell, you can run:
136
137  .. code-block:: console
138
139    $ bugpoint  ... |& tee bugpoint.log
140
141  to get a copy of ``bugpoint``'s output in the file ``bugpoint.log``, as well
142  as on your terminal.
143
144* ``bugpoint`` cannot debug problems with the LLVM linker. If ``bugpoint``
145  crashes before you see its "All input ok" message, you might try ``llvm-link
146  -v`` on the same set of input files. If that also crashes, you may be
147  experiencing a linker bug.
148
149* ``bugpoint`` is useful for proactively finding bugs in LLVM.  Invoking
150  ``bugpoint`` with the ``-find-bugs`` option will cause the list of specified
151  optimizations to be randomized and applied to the program. This process will
152  repeat until a bug is found or the user kills ``bugpoint``.
153
154What to do when bugpoint isn't enough
155=====================================
156
157Sometimes, ``bugpoint`` is not enough. In particular, InstCombine and
158TargetLowering both have visitor structured code with lots of potential
159transformations.  If the process of using bugpoint has left you with still too
160much code to figure out and the problem seems to be in instcombine, the
161following steps may help.  These same techniques are useful with TargetLowering
162as well.
163
164Turn on ``-debug-only=instcombine`` and see which transformations within
165instcombine are firing by selecting out lines with "``IC``" in them.
166
167At this point, you have a decision to make.  Is the number of transformations
168small enough to step through them using a debugger?  If so, then try that.
169
170If there are too many transformations, then a source modification approach may
171be helpful.  In this approach, you can modify the source code of instcombine to
172disable just those transformations that are being performed on your test input
173and perform a binary search over the set of transformations.  One set of places
174to modify are the "``visit*``" methods of ``InstCombiner`` (*e.g.*
175``visitICmpInst``) by adding a "``return false``" as the first line of the
176method.
177
178If that still doesn't remove enough, then change the caller of
179``InstCombiner::DoOneIteration``, ``InstCombiner::runOnFunction`` to limit the
180number of iterations.
181
182You may also find it useful to use "``-stats``" now to see what parts of
183instcombine are firing.  This can guide where to put additional reporting code.
184
185At this point, if the amount of transformations is still too large, then
186inserting code to limit whether or not to execute the body of the code in the
187visit function can be helpful.  Add a static counter which is incremented on
188every invocation of the function.  Then add code which simply returns false on
189desired ranges.  For example:
190
191.. code-block:: c++
192
193
194  static int calledCount = 0;
195  calledCount++;
196  DEBUG(if (calledCount < 212) return false);
197  DEBUG(if (calledCount > 217) return false);
198  DEBUG(if (calledCount == 213) return false);
199  DEBUG(if (calledCount == 214) return false);
200  DEBUG(if (calledCount == 215) return false);
201  DEBUG(if (calledCount == 216) return false);
202  DEBUG(dbgs() << "visitXOR calledCount: " << calledCount << "\n");
203  DEBUG(dbgs() << "I: "; I->dump());
204
205could be added to ``visitXOR`` to limit ``visitXor`` to being applied only to
206calls 212 and 217. This is from an actual test case and raises an important
207point---a simple binary search may not be sufficient, as transformations that
208interact may require isolating more than one call.  In TargetLowering, use
209``return SDNode();`` instead of ``return false;``.
210
211Now that the number of transformations is down to a manageable number, try
212examining the output to see if you can figure out which transformations are
213being done.  If that can be figured out, then do the usual debugging.  If which
214code corresponds to the transformation being performed isn't obvious, set a
215breakpoint after the call count based disabling and step through the code.
216Alternatively, you can use "``printf``" style debugging to report waypoints.
217