1<html>
2<head>
3<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
4<title>7.�Helgrind: a thread error detector</title>
5<link rel="stylesheet" type="text/css" href="vg_basic.css">
6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
7<link rel="home" href="index.html" title="Valgrind Documentation">
8<link rel="up" href="manual.html" title="Valgrind User Manual">
9<link rel="prev" href="cl-manual.html" title="6.�Callgrind: a call-graph generating cache and branch prediction profiler">
10<link rel="next" href="drd-manual.html" title="8.�DRD: a thread error detector">
11</head>
12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
13<div><table class="nav" width="100%" cellspacing="3" cellpadding="3" border="0" summary="Navigation header"><tr>
14<td width="22px" align="center" valign="middle"><a accesskey="p" href="cl-manual.html"><img src="images/prev.png" width="18" height="21" border="0" alt="Prev"></a></td>
15<td width="25px" align="center" valign="middle"><a accesskey="u" href="manual.html"><img src="images/up.png" width="21" height="18" border="0" alt="Up"></a></td>
16<td width="31px" align="center" valign="middle"><a accesskey="h" href="index.html"><img src="images/home.png" width="27" height="20" border="0" alt="Up"></a></td>
17<th align="center" valign="middle">Valgrind User Manual</th>
18<td width="22px" align="center" valign="middle"><a accesskey="n" href="drd-manual.html"><img src="images/next.png" width="18" height="21" border="0" alt="Next"></a></td>
19</tr></table></div>
20<div class="chapter">
21<div class="titlepage"><div><div><h1 class="title">
22<a name="hg-manual"></a>7.�Helgrind: a thread error detector</h1></div></div></div>
23<div class="toc">
24<p><b>Table of Contents</b></p>
25<dl class="toc">
26<dt><span class="sect1"><a href="hg-manual.html#hg-manual.overview">7.1. Overview</a></span></dt>
27<dt><span class="sect1"><a href="hg-manual.html#hg-manual.api-checks">7.2. Detected errors: Misuses of the POSIX pthreads API</a></span></dt>
28<dt><span class="sect1"><a href="hg-manual.html#hg-manual.lock-orders">7.3. Detected errors: Inconsistent Lock Orderings</a></span></dt>
29<dt><span class="sect1"><a href="hg-manual.html#hg-manual.data-races">7.4. Detected errors: Data Races</a></span></dt>
30<dd><dl>
31<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.example">7.4.1. A Simple Data Race</a></span></dt>
32<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.algorithm">7.4.2. Helgrind's Race Detection Algorithm</a></span></dt>
33<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.errmsgs">7.4.3. Interpreting Race Error Messages</a></span></dt>
34</dl></dd>
35<dt><span class="sect1"><a href="hg-manual.html#hg-manual.effective-use">7.5. Hints and Tips for Effective Use of Helgrind</a></span></dt>
36<dt><span class="sect1"><a href="hg-manual.html#hg-manual.options">7.6. Helgrind Command-line Options</a></span></dt>
37<dt><span class="sect1"><a href="hg-manual.html#hg-manual.monitor-commands">7.7. Helgrind Monitor Commands</a></span></dt>
38<dt><span class="sect1"><a href="hg-manual.html#hg-manual.client-requests">7.8. Helgrind Client Requests</a></span></dt>
39<dt><span class="sect1"><a href="hg-manual.html#hg-manual.todolist">7.9. A To-Do List for Helgrind</a></span></dt>
40</dl>
41</div>
42<p>To use this tool, you must specify
43<code class="option">--tool=helgrind</code> on the Valgrind
44command line.</p>
45<div class="sect1">
46<div class="titlepage"><div><div><h2 class="title" style="clear: both">
47<a name="hg-manual.overview"></a>7.1.�Overview</h2></div></div></div>
48<p>Helgrind is a Valgrind tool for detecting synchronisation errors
49in C, C++ and Fortran programs that use the POSIX pthreads
50threading primitives.</p>
51<p>The main abstractions in POSIX pthreads are: a set of threads
52sharing a common address space, thread creation, thread joining,
53thread exit, mutexes (locks), condition variables (inter-thread event
54notifications), reader-writer locks, spinlocks, semaphores and
55barriers.</p>
56<p>Helgrind can detect three classes of errors, which are discussed
57in detail in the next three sections:</p>
58<div class="orderedlist"><ol class="orderedlist" type="1">
59<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.api-checks" title="7.2.�Detected errors: Misuses of the POSIX pthreads API">
60        Misuses of the POSIX pthreads API.</a></p></li>
61<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.lock-orders" title="7.3.�Detected errors: Inconsistent Lock Orderings">
62        Potential deadlocks arising from lock
63        ordering problems.</a></p></li>
64<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.data-races" title="7.4.�Detected errors: Data Races">
65        Data races -- accessing memory without adequate locking
66                      or synchronisation</a>.
67  </p></li>
68</ol></div>
69<p>Problems like these often result in unreproducible,
70timing-dependent crashes, deadlocks and other misbehaviour, and
71can be difficult to find by other means.</p>
72<p>Helgrind is aware of all the pthread abstractions and tracks
73their effects as accurately as it can.  On x86 and amd64 platforms, it
74understands and partially handles implicit locking arising from the
75use of the LOCK instruction prefix.  On PowerPC/POWER and ARM
76platforms, it partially handles implicit locking arising from
77load-linked and store-conditional instruction pairs.
78</p>
79<p>Helgrind works best when your application uses only the POSIX
80pthreads API.  However, if you want to use custom threading
81primitives, you can describe their behaviour to Helgrind using the
82<code class="varname">ANNOTATE_*</code> macros defined
83in <code class="varname">helgrind.h</code>.</p>
84<p>Helgrind also provides <a class="xref" href="manual-core.html#manual-core.xtree" title="2.9.�Execution Trees">Execution Trees</a> memory
85  profiling using the command line
86  option <code class="computeroutput">--xtree-memory</code> and the monitor command
87   <code class="computeroutput">xtmemory</code>.</p>
88<p>Following those is a section containing
89<a class="link" href="hg-manual.html#hg-manual.effective-use" title="7.5.�Hints and Tips for Effective Use of Helgrind">
90hints and tips on how to get the best out of Helgrind.</a>
91</p>
92<p>Then there is a
93<a class="link" href="hg-manual.html#hg-manual.options" title="7.6.�Helgrind Command-line Options">summary of command-line
94options.</a>
95</p>
96<p>Finally, there is
97<a class="link" href="hg-manual.html#hg-manual.todolist" title="7.9.�A To-Do List for Helgrind">a brief summary of areas in which Helgrind
98could be improved.</a>
99</p>
100</div>
101<div class="sect1">
102<div class="titlepage"><div><div><h2 class="title" style="clear: both">
103<a name="hg-manual.api-checks"></a>7.2.�Detected errors: Misuses of the POSIX pthreads API</h2></div></div></div>
104<p>Helgrind intercepts calls to many POSIX pthreads functions, and
105is therefore able to report on various common problems.  Although
106these are unglamourous errors, their presence can lead to undefined
107program behaviour and hard-to-find bugs later on.  The detected errors
108are:</p>
109<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
110<li class="listitem"><p>unlocking an invalid mutex</p></li>
111<li class="listitem"><p>unlocking a not-locked mutex</p></li>
112<li class="listitem"><p>unlocking a mutex held by a different
113                 thread</p></li>
114<li class="listitem"><p>destroying an invalid or a locked mutex</p></li>
115<li class="listitem"><p>recursively locking a non-recursive mutex</p></li>
116<li class="listitem"><p>deallocation of memory that contains a
117                 locked mutex</p></li>
118<li class="listitem"><p>passing mutex arguments to functions expecting
119                 reader-writer lock arguments, and vice
120                 versa</p></li>
121<li class="listitem"><p>when a POSIX pthread function fails with an
122                 error code that must be handled</p></li>
123<li class="listitem"><p>when a thread exits whilst still holding locked
124                 locks</p></li>
125<li class="listitem"><p>calling <code class="function">pthread_cond_wait</code>
126                 with a not-locked mutex, an invalid mutex,
127                 or one locked by a different
128                 thread</p></li>
129<li class="listitem"><p>inconsistent bindings between condition
130                 variables and their associated mutexes</p></li>
131<li class="listitem"><p>invalid or duplicate initialisation of a pthread
132                 barrier</p></li>
133<li class="listitem"><p>initialisation of a pthread barrier on which threads
134                 are still waiting</p></li>
135<li class="listitem"><p>destruction of a pthread barrier object which was
136                 never initialised, or on which threads are still
137                 waiting</p></li>
138<li class="listitem"><p>waiting on an uninitialised pthread
139                 barrier</p></li>
140<li class="listitem"><p>for all of the pthreads functions that Helgrind
141                 intercepts, an error is reported, along with a stack
142                 trace, if the system threading library routine returns
143                 an error code, even if Helgrind itself detected no
144                 error</p></li>
145</ul></div>
146<p>Checks pertaining to the validity of mutexes are generally also
147performed for reader-writer locks.</p>
148<p>Various kinds of this-can't-possibly-happen events are also
149reported.  These usually indicate bugs in the system threading
150library.</p>
151<p>Reported errors always contain a primary stack trace indicating
152where the error was detected.  They may also contain auxiliary stack
153traces giving additional information.  In particular, most errors
154relating to mutexes will also tell you where that mutex first came to
155Helgrind's attention (the "<code class="computeroutput">was first observed
156at</code>" part), so you have a chance of figuring out which
157mutex it is referring to.  For example:</p>
158<pre class="programlisting">
159Thread #1 unlocked a not-locked lock at 0x7FEFFFA90
160   at 0x4C2408D: pthread_mutex_unlock (hg_intercepts.c:492)
161   by 0x40073A: nearly_main (tc09_bad_unlock.c:27)
162   by 0x40079B: main (tc09_bad_unlock.c:50)
163  Lock at 0x7FEFFFA90 was first observed
164   at 0x4C25D01: pthread_mutex_init (hg_intercepts.c:326)
165   by 0x40071F: nearly_main (tc09_bad_unlock.c:23)
166   by 0x40079B: main (tc09_bad_unlock.c:50)
167</pre>
168<p>Helgrind has a way of summarising thread identities, as
169you see here with the text "<code class="computeroutput">Thread
170#1</code>".  This is so that it can speak about threads and
171sets of threads without overwhelming you with details.  See
172<a class="link" href="hg-manual.html#hg-manual.data-races.errmsgs" title="7.4.3.�Interpreting Race Error Messages">below</a>
173for more information on interpreting error messages.</p>
174</div>
175<div class="sect1">
176<div class="titlepage"><div><div><h2 class="title" style="clear: both">
177<a name="hg-manual.lock-orders"></a>7.3.�Detected errors: Inconsistent Lock Orderings</h2></div></div></div>
178<p>In this section, and in general, to "acquire" a lock simply
179means to lock that lock, and to "release" a lock means to unlock
180it.</p>
181<p>Helgrind monitors the order in which threads acquire locks.
182This allows it to detect potential deadlocks which could arise from
183the formation of cycles of locks.  Detecting such inconsistencies is
184useful because, whilst actual deadlocks are fairly obvious, potential
185deadlocks may never be discovered during testing and could later lead
186to hard-to-diagnose in-service failures.</p>
187<p>The simplest example of such a problem is as
188follows.</p>
189<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
190<li class="listitem"><p>Imagine some shared resource R, which, for whatever
191  reason, is guarded by two locks, L1 and L2, which must both be held
192  when R is accessed.</p></li>
193<li class="listitem"><p>Suppose a thread acquires L1, then L2, and proceeds
194  to access R.  The implication of this is that all threads in the
195  program must acquire the two locks in the order first L1 then L2.
196  Not doing so risks deadlock.</p></li>
197<li class="listitem"><p>The deadlock could happen if two threads -- call them
198  T1 and T2 -- both want to access R.  Suppose T1 acquires L1 first,
199  and T2 acquires L2 first.  Then T1 tries to acquire L2, and T2 tries
200  to acquire L1, but those locks are both already held.  So T1 and T2
201  become deadlocked.</p></li>
202</ul></div>
203<p>Helgrind builds a directed graph indicating the order in which
204locks have been acquired in the past.  When a thread acquires a new
205lock, the graph is updated, and then checked to see if it now contains
206a cycle.  The presence of a cycle indicates a potential deadlock involving
207the locks in the cycle.</p>
208<p>In general, Helgrind will choose two locks involved in the cycle
209and show you how their acquisition ordering has become inconsistent.
210It does this by showing the program points that first defined the
211ordering, and the program points which later violated it.  Here is a
212simple example involving just two locks:</p>
213<pre class="programlisting">
214Thread #1: lock order "0x7FF0006D0 before 0x7FF0006A0" violated
215
216Observed (incorrect) order is: acquisition of lock at 0x7FF0006A0
217   at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
218   by 0x400825: main (tc13_laog1.c:23)
219
220 followed by a later acquisition of lock at 0x7FF0006D0
221   at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
222   by 0x400853: main (tc13_laog1.c:24)
223
224Required order was established by acquisition of lock at 0x7FF0006D0
225   at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
226   by 0x40076D: main (tc13_laog1.c:17)
227
228 followed by a later acquisition of lock at 0x7FF0006A0
229   at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
230   by 0x40079B: main (tc13_laog1.c:18)
231</pre>
232<p>When there are more than two locks in the cycle, the error is
233equally serious.  However, at present Helgrind does not show the locks
234involved, sometimes because that information is not available, but
235also so as to avoid flooding you with information.  For example, a
236naive implementation of the famous Dining Philosophers problem
237involves a cycle of five locks
238(see <code class="computeroutput">helgrind/tests/tc14_laog_dinphils.c</code>).
239In this case Helgrind has detected that all 5 philosophers could
240simultaneously pick up their left fork and then deadlock whilst
241waiting to pick up their right forks.</p>
242<pre class="programlisting">
243Thread #6: lock order "0x80499A0 before 0x8049A00" violated
244
245Observed (incorrect) order is: acquisition of lock at 0x8049A00
246   at 0x40085BC: pthread_mutex_lock (hg_intercepts.c:495)
247   by 0x80485B4: dine (tc14_laog_dinphils.c:18)
248   by 0x400BDA4: mythread_wrapper (hg_intercepts.c:219)
249   by 0x39B924: start_thread (pthread_create.c:297)
250   by 0x2F107D: clone (clone.S:130)
251
252 followed by a later acquisition of lock at 0x80499A0
253   at 0x40085BC: pthread_mutex_lock (hg_intercepts.c:495)
254   by 0x80485CD: dine (tc14_laog_dinphils.c:19)
255   by 0x400BDA4: mythread_wrapper (hg_intercepts.c:219)
256   by 0x39B924: start_thread (pthread_create.c:297)
257   by 0x2F107D: clone (clone.S:130)
258</pre>
259</div>
260<div class="sect1">
261<div class="titlepage"><div><div><h2 class="title" style="clear: both">
262<a name="hg-manual.data-races"></a>7.4.�Detected errors: Data Races</h2></div></div></div>
263<p>A data race happens, or could happen, when two threads access a
264shared memory location without using suitable locks or other
265synchronisation to ensure single-threaded access.  Such missing
266locking can cause obscure timing dependent bugs.  Ensuring programs
267are race-free is one of the central difficulties of threaded
268programming.</p>
269<p>Reliably detecting races is a difficult problem, and most
270of Helgrind's internals are devoted to dealing with it.
271We begin with a simple example.</p>
272<div class="sect2">
273<div class="titlepage"><div><div><h3 class="title">
274<a name="hg-manual.data-races.example"></a>7.4.1.�A Simple Data Race</h3></div></div></div>
275<p>About the simplest possible example of a race is as follows.  In
276this program, it is impossible to know what the value
277of <code class="computeroutput">var</code> is at the end of the program.
278Is it 2 ?  Or 1 ?</p>
279<pre class="programlisting">
280#include &lt;pthread.h&gt;
281
282int var = 0;
283
284void* child_fn ( void* arg ) {
285   var++; /* Unprotected relative to parent */ /* this is line 6 */
286   return NULL;
287}
288
289int main ( void ) {
290   pthread_t child;
291   pthread_create(&amp;child, NULL, child_fn, NULL);
292   var++; /* Unprotected relative to child */ /* this is line 13 */
293   pthread_join(child, NULL);
294   return 0;
295}
296</pre>
297<p>The problem is there is nothing to
298stop <code class="varname">var</code> being updated simultaneously
299by both threads.  A correct program would
300protect <code class="varname">var</code> with a lock of type
301<code class="function">pthread_mutex_t</code>, which is acquired
302before each access and released afterwards.  Helgrind's output for
303this program is:</p>
304<pre class="programlisting">
305Thread #1 is the program's root thread
306
307Thread #2 was created
308   at 0x511C08E: clone (in /lib64/libc-2.8.so)
309   by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
310   by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
311   by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
312   by 0x400605: main (simple_race.c:12)
313
314Possible data race during read of size 4 at 0x601038 by thread #1
315Locks held: none
316   at 0x400606: main (simple_race.c:13)
317
318This conflicts with a previous write of size 4 by thread #2
319Locks held: none
320   at 0x4005DC: child_fn (simple_race.c:6)
321   by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
322   by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
323   by 0x511C0CC: clone (in /lib64/libc-2.8.so)
324
325Location 0x601038 is 0 bytes inside global var "var"
326declared at simple_race.c:3
327</pre>
328<p>This is quite a lot of detail for an apparently simple error.
329The last clause is the main error message.  It says there is a race as
330a result of a read of size 4 (bytes), at 0x601038, which is the
331address of <code class="computeroutput">var</code>, happening in
332function <code class="computeroutput">main</code> at line 13 in the
333program.</p>
334<p>Two important parts of the message are:</p>
335<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
336<li class="listitem">
337<p>Helgrind shows two stack traces for the error, not one.  By
338   definition, a race involves two different threads accessing the
339   same location in such a way that the result depends on the relative
340   speeds of the two threads.</p>
341<p>
342   The first stack trace follows the text "<code class="computeroutput">Possible
343   data race during read of size 4 ...</code>" and the
344   second trace follows the text "<code class="computeroutput">This conflicts with
345   a previous write of size 4 ...</code>".  Helgrind is
346   usually able to show both accesses involved in a race.  At least
347   one of these will be a write (since two concurrent, unsynchronised
348   reads are harmless), and they will of course be from different
349   threads.</p>
350<p>By examining your program at the two locations, you should be
351   able to get at least some idea of what the root cause of the
352   problem is.  For each location, Helgrind shows the set of locks
353   held at the time of the access.  This often makes it clear which
354   thread, if any, failed to take a required lock.  In this example
355   neither thread holds a lock during the access.</p>
356</li>
357<li class="listitem">
358<p>For races which occur on global or stack variables, Helgrind
359   tries to identify the name and defining point of the variable.
360   Hence the text "<code class="computeroutput">Location 0x601038 is 0 bytes inside
361   global var "var" declared at simple_race.c:3</code>".</p>
362<p>Showing names of stack and global variables carries no
363   run-time overhead once Helgrind has your program up and running.
364   However, it does require Helgrind to spend considerable extra time
365   and memory at program startup to read the relevant debug info.
366   Hence this facility is disabled by default.  To enable it, you need
367   to give the <code class="varname">--read-var-info=yes</code> option to
368   Helgrind.</p>
369</li>
370</ul></div>
371<p>The following section explains Helgrind's race detection
372algorithm in more detail.</p>
373</div>
374<div class="sect2">
375<div class="titlepage"><div><div><h3 class="title">
376<a name="hg-manual.data-races.algorithm"></a>7.4.2.�Helgrind's Race Detection Algorithm</h3></div></div></div>
377<p>Most programmers think about threaded programming in terms of
378the basic functionality provided by the threading library (POSIX
379Pthreads): thread creation, thread joining, locks, condition
380variables, semaphores and barriers.</p>
381<p>The effect of using these functions is to impose
382constraints upon the order in which memory accesses can
383happen.  This implied ordering is generally known as the
384"happens-before relation".  Once you understand the happens-before
385relation, it is easy to see how Helgrind finds races in your code.
386Fortunately, the happens-before relation is itself easy to understand,
387and is by itself a useful tool for reasoning about the behaviour of
388parallel programs.  We now introduce it using a simple example.</p>
389<p>Consider first the following buggy program:</p>
390<pre class="programlisting">
391Parent thread:                         Child thread:
392
393int var;
394
395// create child thread
396pthread_create(...)
397var = 20;                              var = 10;
398                                       exit
399
400// wait for child
401pthread_join(...)
402printf("%d\n", var);
403</pre>
404<p>The parent thread creates a child.  Both then write different
405values to some variable <code class="computeroutput">var</code>, and the
406parent then waits for the child to exit.</p>
407<p>What is the value of <code class="computeroutput">var</code> at the
408end of the program, 10 or 20?  We don't know.  The program is
409considered buggy (it has a race) because the final value
410of <code class="computeroutput">var</code> depends on the relative rates
411of progress of the parent and child threads.  If the parent is fast
412and the child is slow, then the child's assignment may happen later,
413so the final value will be 10; and vice versa if the child is faster
414than the parent.</p>
415<p>The relative rates of progress of parent vs child is not something
416the programmer can control, and will often change from run to run.
417It depends on factors such as the load on the machine, what else is
418running, the kernel's scheduling strategy, and many other factors.</p>
419<p>The obvious fix is to use a lock to
420protect <code class="computeroutput">var</code>.  It is however
421instructive to consider a somewhat more abstract solution, which is to
422send a message from one thread to the other:</p>
423<pre class="programlisting">
424Parent thread:                         Child thread:
425
426int var;
427
428// create child thread
429pthread_create(...)
430var = 20;
431// send message to child
432                                       // wait for message to arrive
433                                       var = 10;
434                                       exit
435
436// wait for child
437pthread_join(...)
438printf("%d\n", var);
439</pre>
440<p>Now the program reliably prints "10", regardless of the speed of
441the threads.  Why?  Because the child's assignment cannot happen until
442after it receives the message.  And the message is not sent until
443after the parent's assignment is done.</p>
444<p>The message transmission creates a "happens-before" dependency
445between the two assignments: <code class="computeroutput">var = 20;</code>
446must now happen-before <code class="computeroutput">var = 10;</code>.
447And so there is no longer a race
448on <code class="computeroutput">var</code>.
449</p>
450<p>Note that it's not significant that the parent sends a message
451to the child.  Sending a message from the child (after its assignment)
452to the parent (before its assignment) would also fix the problem, causing
453the program to reliably print "20".</p>
454<p>Helgrind's algorithm is (conceptually) very simple.  It monitors all
455accesses to memory locations.  If a location -- in this example,
456<code class="computeroutput">var</code>,
457is accessed by two different threads, Helgrind checks to see if the
458two accesses are ordered by the happens-before relation.  If so,
459that's fine; if not, it reports a race.</p>
460<p>It is important to understand that the happens-before relation
461creates only a partial ordering, not a total ordering.  An example of
462a total ordering is comparison of numbers: for any two numbers
463<code class="computeroutput">x</code> and
464<code class="computeroutput">y</code>, either
465<code class="computeroutput">x</code> is less than, equal to, or greater
466than
467<code class="computeroutput">y</code>.  A partial ordering is like a
468total ordering, but it can also express the concept that two elements
469are neither equal, less or greater, but merely unordered with respect
470to each other.</p>
471<p>In the fixed example above, we say that
472<code class="computeroutput">var = 20;</code> "happens-before"
473<code class="computeroutput">var = 10;</code>.  But in the original
474version, they are unordered: we cannot say that either happens-before
475the other.</p>
476<p>What does it mean to say that two accesses from different
477threads are ordered by the happens-before relation?  It means that
478there is some chain of inter-thread synchronisation operations which
479cause those accesses to happen in a particular order, irrespective of
480the actual rates of progress of the individual threads.  This is a
481required property for a reliable threaded program, which is why
482Helgrind checks for it.</p>
483<p>The happens-before relations created by standard threading
484primitives are as follows:</p>
485<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
486<li class="listitem"><p>When a mutex is unlocked by thread T1 and later (or
487  immediately) locked by thread T2, then the memory accesses in T1
488  prior to the unlock must happen-before those in T2 after it acquires
489  the lock.</p></li>
490<li class="listitem"><p>The same idea applies to reader-writer locks,
491  although with some complication so as to allow correct handling of
492  reads vs writes.</p></li>
493<li class="listitem"><p>When a condition variable (CV) is signalled on by
494  thread T1 and some other thread T2 is thereby released from a wait
495  on the same CV, then the memory accesses in T1 prior to the
496  signalling must happen-before those in T2 after it returns from the
497  wait.  If no thread was waiting on the CV then there is no
498  effect.</p></li>
499<li class="listitem"><p>If instead T1 broadcasts on a CV, then all of the
500  waiting threads, rather than just one of them, acquire a
501  happens-before dependency on the broadcasting thread at the point it
502  did the broadcast.</p></li>
503<li class="listitem"><p>A thread T2 that continues after completing sem_wait
504  on a semaphore that thread T1 posts on, acquires a happens-before
505  dependence on the posting thread, a bit like dependencies caused
506  mutex unlock-lock pairs.  However, since a semaphore can be posted
507  on many times, it is unspecified from which of the post calls the
508  wait call gets its happens-before dependency.</p></li>
509<li class="listitem"><p>For a group of threads T1 .. Tn which arrive at a
510  barrier and then move on, each thread after the call has a
511  happens-after dependency from all threads before the
512  barrier.</p></li>
513<li class="listitem"><p>A newly-created child thread acquires an initial
514  happens-after dependency on the point where its parent created it.
515  That is, all memory accesses performed by the parent prior to
516  creating the child are regarded as happening-before all the accesses
517  of the child.</p></li>
518<li class="listitem"><p>Similarly, when an exiting thread is reaped via a
519  call to <code class="function">pthread_join</code>, once the call returns, the
520  reaping thread acquires a happens-after dependency relative to all memory
521  accesses made by the exiting thread.</p></li>
522</ul></div>
523<p>In summary: Helgrind intercepts the above listed events, and builds a
524directed acyclic graph represented the collective happens-before
525dependencies.  It also monitors all memory accesses.</p>
526<p>If a location is accessed by two different threads, but Helgrind
527cannot find any path through the happens-before graph from one access
528to the other, then it reports a race.</p>
529<p>There are a couple of caveats:</p>
530<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
531<li class="listitem"><p>Helgrind doesn't check for a race in the case where
532  both accesses are reads.  That would be silly, since concurrent
533  reads are harmless.</p></li>
534<li class="listitem"><p>Two accesses are considered to be ordered by the
535  happens-before dependency even through arbitrarily long chains of
536  synchronisation events.  For example, if T1 accesses some location
537  L, and then <code class="function">pthread_cond_signals</code> T2, which later
538  <code class="function">pthread_cond_signals</code> T3, which then accesses L, then
539  a suitable happens-before dependency exists between the first and second
540  accesses, even though it involves two different inter-thread
541  synchronisation events.</p></li>
542</ul></div>
543</div>
544<div class="sect2">
545<div class="titlepage"><div><div><h3 class="title">
546<a name="hg-manual.data-races.errmsgs"></a>7.4.3.�Interpreting Race Error Messages</h3></div></div></div>
547<p>Helgrind's race detection algorithm collects a lot of
548information, and tries to present it in a helpful way when a race is
549detected.  Here's an example:</p>
550<pre class="programlisting">
551Thread #2 was created
552   at 0x511C08E: clone (in /lib64/libc-2.8.so)
553   by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
554   by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
555   by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
556   by 0x4008F2: main (tc21_pthonce.c:86)
557
558Thread #3 was created
559   at 0x511C08E: clone (in /lib64/libc-2.8.so)
560   by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
561   by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
562   by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
563   by 0x4008F2: main (tc21_pthonce.c:86)
564
565Possible data race during read of size 4 at 0x601070 by thread #3
566Locks held: none
567   at 0x40087A: child (tc21_pthonce.c:74)
568   by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
569   by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
570   by 0x511C0CC: clone (in /lib64/libc-2.8.so)
571
572This conflicts with a previous write of size 4 by thread #2
573Locks held: none
574   at 0x400883: child (tc21_pthonce.c:74)
575   by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
576   by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
577   by 0x511C0CC: clone (in /lib64/libc-2.8.so)
578
579Location 0x601070 is 0 bytes inside local var "unprotected2"
580declared at tc21_pthonce.c:51, in frame #0 of thread 3
581</pre>
582<p>Helgrind first announces the creation points of any threads
583referenced in the error message.  This is so it can speak concisely
584about threads without repeatedly printing their creation point call
585stacks.  Each thread is only ever announced once, the first time it
586appears in any Helgrind error message.</p>
587<p>The main error message begins at the text
588"<code class="computeroutput">Possible data race during read</code>".  At
589the start is information you would expect to see -- address and size
590of the racing access, whether a read or a write, and the call stack at
591the point it was detected.</p>
592<p>A second call stack is presented starting at the text
593"<code class="computeroutput">This conflicts with a previous
594write</code>".  This shows a previous access which also
595accessed the stated address, and which is believed to be racing
596against the access in the first call stack. Note that this second
597call stack is limited to a maximum of 8 entries to limit the
598memory usage.</p>
599<p>Finally, Helgrind may attempt to give a description of the
600raced-on address in source level terms.  In this example, it
601identifies it as a local variable, shows its name, declaration point,
602and in which frame (of the first call stack) it lives.  Note that this
603information is only shown when <code class="varname">--read-var-info=yes</code>
604is specified on the command line.  That's because reading the DWARF3
605debug information in enough detail to capture variable type and
606location information makes Helgrind much slower at startup, and also
607requires considerable amounts of memory, for large programs.
608</p>
609<p>Once you have your two call stacks, how do you find the root
610cause of the race?</p>
611<p>The first thing to do is examine the source locations referred
612to by each call stack.  They should both show an access to the same
613location, or variable.</p>
614<p>Now figure out how how that location should have been made
615thread-safe:</p>
616<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
617<li class="listitem"><p>Perhaps the location was intended to be protected by
618  a mutex?  If so, you need to lock and unlock the mutex at both
619  access points, even if one of the accesses is reported to be a read.
620  Did you perhaps forget the locking at one or other of the accesses?
621  To help you do this, Helgrind shows the set of locks held by each
622  threads at the time they accessed the raced-on location.</p></li>
623<li class="listitem">
624<p>Alternatively, perhaps you intended to use a some
625  other scheme to make it safe, such as signalling on a condition
626  variable.  In all such cases, try to find a synchronisation event
627  (or a chain thereof) which separates the earlier-observed access (as
628  shown in the second call stack) from the later-observed access (as
629  shown in the first call stack).  In other words, try to find
630  evidence that the earlier access "happens-before" the later access.
631  See the previous subsection for an explanation of the happens-before
632  relation.</p>
633<p>
634  The fact that Helgrind is reporting a race means it did not observe
635  any happens-before relation between the two accesses.  If
636  Helgrind is working correctly, it should also be the case that you
637  also cannot find any such relation, even on detailed inspection
638  of the source code.  Hopefully, though, your inspection of the code
639  will show where the missing synchronisation operation(s) should have
640  been.</p>
641</li>
642</ul></div>
643</div>
644</div>
645<div class="sect1">
646<div class="titlepage"><div><div><h2 class="title" style="clear: both">
647<a name="hg-manual.effective-use"></a>7.5.�Hints and Tips for Effective Use of Helgrind</h2></div></div></div>
648<p>Helgrind can be very helpful in finding and resolving
649threading-related problems.  Like all sophisticated tools, it is most
650effective when you understand how to play to its strengths.</p>
651<p>Helgrind will be less effective when you merely throw an
652existing threaded program at it and try to make sense of any reported
653errors.  It will be more effective if you design threaded programs
654from the start in a way that helps Helgrind verify correctness.  The
655same is true for finding memory errors with Memcheck, but applies more
656here, because thread checking is a harder problem.  Consequently it is
657much easier to write a correct program for which Helgrind falsely
658reports (threading) errors than it is to write a correct program for
659which Memcheck falsely reports (memory) errors.</p>
660<p>With that in mind, here are some tips, listed most important first,
661for getting reliable results and avoiding false errors.  The first two
662are critical.  Any violations of them will swamp you with huge numbers
663of false data-race errors.</p>
664<div class="orderedlist"><ol class="orderedlist" type="1">
665<li class="listitem">
666<p>Make sure your application, and all the libraries it uses,
667    use the POSIX threading primitives.  Helgrind needs to be able to
668    see all events pertaining to thread creation, exit, locking and
669    other synchronisation events.  To do so it intercepts many POSIX
670    pthreads functions.</p>
671<p>Do not roll your own threading primitives (mutexes, etc)
672    from combinations of the Linux futex syscall, atomic counters, etc.
673    These throw Helgrind's internal what's-going-on models
674    way off course and will give bogus results.</p>
675<p>Also, do not reimplement existing POSIX abstractions using
676    other POSIX abstractions.  For example, don't build your own
677    semaphore routines or reader-writer locks from POSIX mutexes and
678    condition variables.  Instead use POSIX reader-writer locks and
679    semaphores directly, since Helgrind supports them directly.</p>
680<p>Helgrind directly supports the following POSIX threading
681    abstractions: mutexes, reader-writer locks, condition variables
682    (but see below), semaphores and barriers.  Currently spinlocks
683    are not supported, although they could be in future.</p>
684<p>At the time of writing, the following popular Linux packages
685    are known to implement their own threading primitives:</p>
686<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
687<li class="listitem"><p>Qt version 4.X.  Qt 3.X is harmless in that it
688      only uses POSIX pthreads primitives.  Unfortunately Qt 4.X
689      has its own implementation of mutexes (QMutex) and thread reaping.
690      Helgrind 3.4.x contains direct support
691      for Qt 4.X threading, which is experimental but is believed to
692      work fairly well.  A side effect of supporting Qt 4 directly is
693      that Helgrind can be used to debug KDE4 applications.  As this
694      is an experimental feature, we would particularly appreciate
695      feedback from folks who have used Helgrind to successfully debug
696      Qt 4 and/or KDE4 applications.</p></li>
697<li class="listitem">
698<p>Runtime support library for GNU OpenMP (part of
699      GCC), at least for GCC versions 4.2 and 4.3.  The GNU OpenMP runtime
700      library (<code class="filename">libgomp.so</code>) constructs its own
701      synchronisation primitives using combinations of atomic memory
702      instructions and the futex syscall, which causes total chaos since in
703      Helgrind since it cannot "see" those.</p>
704<p>Fortunately, this can be solved using a configuration-time
705      option (for GCC).  Rebuild GCC from source, and configure using
706      <code class="varname">--disable-linux-futex</code>.
707      This makes libgomp.so use the standard
708      POSIX threading primitives instead.  Note that this was tested
709      using GCC 4.2.3 and has not been re-tested using more recent GCC
710      versions.  We would appreciate hearing about any successes or
711      failures with more recent versions.</p>
712</li>
713</ul></div>
714<p>If you must implement your own threading primitives, there
715      are a set of client request macros
716      in <code class="computeroutput">helgrind.h</code> to help you
717      describe your primitives to Helgrind.  You should be able to
718      mark up mutexes, condition variables, etc, without difficulty.
719    </p>
720<p>
721      It is also possible to mark up the effects of thread-safe
722      reference counting using the
723      <code class="computeroutput">ANNOTATE_HAPPENS_BEFORE</code>,
724      <code class="computeroutput">ANNOTATE_HAPPENS_AFTER</code> and
725      <code class="computeroutput">ANNOTATE_HAPPENS_BEFORE_FORGET_ALL</code>,
726      macros.  Thread-safe reference counting using an atomically
727      incremented/decremented refcount variable causes Helgrind
728      problems because a one-to-zero transition of the reference count
729      means the accessing thread has exclusive ownership of the
730      associated resource (normally, a C++ object) and can therefore
731      access it (normally, to run its destructor) without locking.
732      Helgrind doesn't understand this, and markup is essential to
733      avoid false positives.
734    </p>
735<p>
736      Here are recommended guidelines for marking up thread safe
737      reference counting in C++.  You only need to mark up your
738      release methods -- the ones which decrement the reference count.
739      Given a class like this:
740    </p>
741<pre class="programlisting">
742class MyClass {
743   unsigned int mRefCount;
744
745   void Release ( void ) {
746      unsigned int newCount = atomic_decrement(&amp;mRefCount);
747      if (newCount == 0) {
748         delete this;
749      }
750   }
751}
752</pre>
753<p>
754     the release method should be marked up as follows:
755   </p>
756<pre class="programlisting">
757   void Release ( void ) {
758      unsigned int newCount = atomic_decrement(&amp;mRefCount);
759      if (newCount == 0) {
760         ANNOTATE_HAPPENS_AFTER(&amp;mRefCount);
761         ANNOTATE_HAPPENS_BEFORE_FORGET_ALL(&amp;mRefCount);
762         delete this;
763      } else {
764         ANNOTATE_HAPPENS_BEFORE(&amp;mRefCount);
765      }
766   }
767</pre>
768<p>
769      There are a number of complex, mostly-theoretical objections to
770      this scheme.  From a theoretical standpoint it appears to be
771      impossible to devise a markup scheme which is completely correct
772      in the sense of guaranteeing to remove all false races.  The
773      proposed scheme however works well in practice.
774    </p>
775</li>
776<li class="listitem">
777<p>Avoid memory recycling.  If you can't avoid it, you must use
778    tell Helgrind what is going on via the
779    <code class="function">VALGRIND_HG_CLEAN_MEMORY</code> client request (in
780    <code class="computeroutput">helgrind.h</code>).</p>
781<p>Helgrind is aware of standard heap memory allocation and
782    deallocation that occurs via
783    <code class="function">malloc</code>/<code class="function">free</code>/<code class="function">new</code>/<code class="function">delete</code>
784    and from entry and exit of stack frames.  In particular, when memory is
785    deallocated via <code class="function">free</code>, <code class="function">delete</code>,
786    or function exit, Helgrind considers that memory clean, so when it is
787    eventually reallocated, its history is irrelevant.</p>
788<p>However, it is common practice to implement memory recycling
789    schemes.  In these, memory to be freed is not handed to
790    <code class="function">free</code>/<code class="function">delete</code>, but instead put
791    into a pool of free buffers to be handed out again as required.  The
792    problem is that Helgrind has no
793    way to know that such memory is logically no longer in use, and
794    its history is irrelevant.  Hence you must make that explicit,
795    using the <code class="function">VALGRIND_HG_CLEAN_MEMORY</code> client request
796    to specify the relevant address ranges.  It's easiest to put these
797    requests into the pool manager code, and use them either when memory is
798    returned to the pool, or is allocated from it.</p>
799</li>
800<li class="listitem">
801<p>Avoid POSIX condition variables.  If you can, use POSIX
802    semaphores (<code class="function">sem_t</code>, <code class="function">sem_post</code>,
803    <code class="function">sem_wait</code>) to do inter-thread event signalling.
804    Semaphores with an initial value of zero are particularly useful for
805    this.</p>
806<p>Helgrind only partially correctly handles POSIX condition
807    variables.  This is because Helgrind can see inter-thread
808    dependencies between a <code class="function">pthread_cond_wait</code> call and a
809    <code class="function">pthread_cond_signal</code>/<code class="function">pthread_cond_broadcast</code>
810    call only if the waiting thread actually gets to the rendezvous first
811    (so that it actually calls
812    <code class="function">pthread_cond_wait</code>).  It can't see dependencies
813    between the threads if the signaller arrives first.  In the latter case,
814    POSIX guidelines imply that the associated boolean condition still
815    provides an inter-thread synchronisation event, but one which is
816    invisible to Helgrind.</p>
817<p>The result of Helgrind missing some inter-thread
818    synchronisation events is to cause it to report false positives.
819    </p>
820<p>The root cause of this synchronisation lossage is
821    particularly hard to understand, so an example is helpful.  It was
822    discussed at length by Arndt Muehlenfeld ("Runtime Race Detection
823    in Multi-Threaded Programs", Dissertation, TU Graz, Austria).  The
824    canonical POSIX-recommended usage scheme for condition variables
825    is as follows:</p>
826<pre class="programlisting">
827b   is a Boolean condition, which is False most of the time
828cv  is a condition variable
829mx  is its associated mutex
830
831Signaller:                             Waiter:
832
833lock(mx)                               lock(mx)
834b = True                               while (b == False)
835signal(cv)                                wait(cv,mx)
836unlock(mx)                             unlock(mx)
837</pre>
838<p>Assume <code class="computeroutput">b</code> is False most of
839    the time.  If the waiter arrives at the rendezvous first, it
840    enters its while-loop, waits for the signaller to signal, and
841    eventually proceeds.  Helgrind sees the signal, notes the
842    dependency, and all is well.</p>
843<p>If the signaller arrives
844    first, <code class="computeroutput">b</code> is set to true, and the
845    signal disappears into nowhere.  When the waiter later arrives, it
846    does not enter its while-loop and simply carries on.  But even in
847    this case, the waiter code following the while-loop cannot execute
848    until the signaller sets <code class="computeroutput">b</code> to
849    True.  Hence there is still the same inter-thread dependency, but
850    this time it is through an arbitrary in-memory condition, and
851    Helgrind cannot see it.</p>
852<p>By comparison, Helgrind's detection of inter-thread
853    dependencies caused by semaphore operations is believed to be
854    exactly correct.</p>
855<p>As far as I know, a solution to this problem that does not
856    require source-level annotation of condition-variable wait loops
857    is beyond the current state of the art.</p>
858</li>
859<li class="listitem"><p>Make sure you are using a supported Linux distribution.  At
860    present, Helgrind only properly supports glibc-2.3 or later.  This
861    in turn means we only support glibc's NPTL threading
862    implementation.  The old LinuxThreads implementation is not
863    supported.</p></li>
864<li class="listitem"><p>If your application is using thread local variables,
865    helgrind might report false positive race conditions on these
866    variables, despite being very probably race free.  On Linux, you can
867    use <code class="option">--sim-hints=deactivate-pthread-stack-cache-via-hack</code>
868    to avoid such false positive error messages
869    (see <a class="xref" href="manual-core.html#opt.sim-hints">--sim-hints</a>).
870    </p></li>
871<li class="listitem">
872<p>Round up all finished threads using
873    <code class="function">pthread_join</code>.  Avoid
874    detaching threads: don't create threads in the detached state, and
875    don't call <code class="function">pthread_detach</code> on existing threads.</p>
876<p>Using <code class="function">pthread_join</code> to round up finished
877    threads provides a clear synchronisation point that both Helgrind and
878    programmers can see.  If you don't call
879    <code class="function">pthread_join</code> on a thread, Helgrind has no way to
880    know when it finishes, relative to any
881    significant synchronisation points for other threads in the program.  So
882    it assumes that the thread lingers indefinitely and can potentially
883    interfere indefinitely with the memory state of the program.  It
884    has every right to assume that -- after all, it might really be
885    the case that, for scheduling reasons, the exiting thread did run
886    very slowly in the last stages of its life.</p>
887</li>
888<li class="listitem">
889<p>Perform thread debugging (with Helgrind) and memory
890    debugging (with Memcheck) together.</p>
891<p>Helgrind tracks the state of memory in detail, and memory
892    management bugs in the application are liable to cause confusion.
893    In extreme cases, applications which do many invalid reads and
894    writes (particularly to freed memory) have been known to crash
895    Helgrind.  So, ideally, you should make your application
896    Memcheck-clean before using Helgrind.</p>
897<p>It may be impossible to make your application Memcheck-clean
898    unless you first remove threading bugs.  In particular, it may be
899    difficult to remove all reads and writes to freed memory in
900    multithreaded C++ destructor sequences at program termination.
901    So, ideally, you should make your application Helgrind-clean
902    before using Memcheck.</p>
903<p>Since this circularity is obviously unresolvable, at least
904    bear in mind that Memcheck and Helgrind are to some extent
905    complementary, and you may need to use them together.</p>
906</li>
907<li class="listitem">
908<p>POSIX requires that implementations of standard I/O
909    (<code class="function">printf</code>, <code class="function">fprintf</code>,
910    <code class="function">fwrite</code>, <code class="function">fread</code>, etc) are thread
911    safe.  Unfortunately GNU libc implements this by using internal locking
912    primitives that Helgrind is unable to intercept.  Consequently Helgrind
913    generates many false race reports when you use these functions.</p>
914<p>Helgrind attempts to hide these errors using the standard
915    Valgrind error-suppression mechanism.  So, at least for simple
916    test cases, you don't see any.  Nevertheless, some may slip
917    through.  Just something to be aware of.</p>
918</li>
919<li class="listitem">
920<p>Helgrind's error checks do not work properly inside the
921    system threading library itself
922    (<code class="computeroutput">libpthread.so</code>), and it usually
923    observes large numbers of (false) errors in there.  Valgrind's
924    suppression system then filters these out, so you should not see
925    them.</p>
926<p>If you see any race errors reported
927    where <code class="computeroutput">libpthread.so</code> or
928    <code class="computeroutput">ld.so</code> is the object associated
929    with the innermost stack frame, please file a bug report at
930    <a class="ulink" href="http://www.valgrind.org/" target="_top">http://www.valgrind.org/</a>.
931    </p>
932</li>
933</ol></div>
934</div>
935<div class="sect1">
936<div class="titlepage"><div><div><h2 class="title" style="clear: both">
937<a name="hg-manual.options"></a>7.6.�Helgrind Command-line Options</h2></div></div></div>
938<p>The following end-user options are available:</p>
939<div class="variablelist">
940<a name="hg.opts.list"></a><dl class="variablelist">
941<dt>
942<a name="opt.free-is-write"></a><span class="term">
943      <code class="option">--free-is-write=no|yes
944      [default: no] </code>
945    </span>
946</dt>
947<dd>
948<p>When enabled (not the default), Helgrind treats freeing of
949        heap memory as if the memory was written immediately before
950        the free.  This exposes races where memory is referenced by
951        one thread, and freed by another, but there is no observable
952        synchronisation event to ensure that the reference happens
953        before the free.
954      </p>
955<p>This functionality is new in Valgrind 3.7.0, and is
956        regarded as experimental.  It is not enabled by default
957        because its interaction with custom memory allocators is not
958        well understood at present.  User feedback is welcomed.
959      </p>
960</dd>
961<dt>
962<a name="opt.track-lockorders"></a><span class="term">
963      <code class="option">--track-lockorders=no|yes
964      [default: yes] </code>
965    </span>
966</dt>
967<dd><p>When enabled (the default), Helgrind performs lock order
968      consistency checking.  For some buggy programs, the large number
969      of lock order errors reported can become annoying, particularly
970      if you're only interested in race errors.  You may therefore find
971      it helpful to disable lock order checking.</p></dd>
972<dt>
973<a name="opt.history-level"></a><span class="term">
974      <code class="option">--history-level=none|approx|full
975      [default: full] </code>
976    </span>
977</dt>
978<dd>
979<p><code class="option">--history-level=full</code> (the default) causes
980        Helgrind collects enough information about "old" accesses that
981        it can produce two stack traces in a race report -- both the
982        stack trace for the current access, and the trace for the
983        older, conflicting access. To limit memory usage, "old" accesses
984        stack traces are limited to a maximum of 8 entries, even if
985        <code class="option">--num-callers</code> value is bigger.</p>
986<p>Collecting such information is expensive in both speed and
987        memory, particularly for programs that do many inter-thread
988        synchronisation events (locks, unlocks, etc).  Without such
989        information, it is more difficult to track down the root
990        causes of races.  Nonetheless, you may not need it in
991        situations where you just want to check for the presence or
992        absence of races, for example, when doing regression testing
993        of a previously race-free program.</p>
994<p><code class="option">--history-level=none</code> is the opposite
995        extreme.  It causes Helgrind not to collect any information
996        about previous accesses.  This can be dramatically faster
997        than <code class="option">--history-level=full</code>.</p>
998<p><code class="option">--history-level=approx</code> provides a
999        compromise between these two extremes.  It causes Helgrind to
1000        show a full trace for the later access, and approximate
1001        information regarding the earlier access.  This approximate
1002        information consists of two stacks, and the earlier access is
1003        guaranteed to have occurred somewhere between program points
1004        denoted by the two stacks. This is not as useful as showing
1005        the exact stack for the previous access
1006        (as <code class="option">--history-level=full</code> does), but it is
1007        better than nothing, and it is almost as fast as
1008        <code class="option">--history-level=none</code>.</p>
1009</dd>
1010<dt>
1011<a name="opt.conflict-cache-size"></a><span class="term">
1012      <code class="option">--conflict-cache-size=N
1013      [default: 1000000] </code>
1014    </span>
1015</dt>
1016<dd>
1017<p>This flag only has any effect
1018        at <code class="option">--history-level=full</code>.</p>
1019<p>Information about "old" conflicting accesses is stored in
1020        a cache of limited size, with LRU-style management.  This is
1021        necessary because it isn't practical to store a stack trace
1022        for every single memory access made by the program.
1023        Historical information on not recently accessed locations is
1024        periodically discarded, to free up space in the cache.</p>
1025<p>This option controls the size of the cache, in terms of the
1026        number of different memory addresses for which
1027        conflicting access information is stored.  If you find that
1028        Helgrind is showing race errors with only one stack instead of
1029        the expected two stacks, try increasing this value.</p>
1030<p>The minimum value is 10,000 and the maximum is 30,000,000
1031        (thirty times the default value).  Increasing the value by 1
1032        increases Helgrind's memory requirement by very roughly 100
1033        bytes, so the maximum value will easily eat up three extra
1034        gigabytes or so of memory.</p>
1035</dd>
1036<dt>
1037<a name="opt.check-stack-refs"></a><span class="term">
1038      <code class="option">--check-stack-refs=no|yes
1039      [default: yes] </code>
1040    </span>
1041</dt>
1042<dd><p>
1043        By default Helgrind checks all data memory accesses made by your
1044        program.  This flag enables you to skip checking for accesses
1045        to thread stacks (local variables).  This can improve
1046        performance, but comes at the cost of missing races on
1047        stack-allocated data.
1048      </p></dd>
1049<dt>
1050<a name="opt.ignore-thread-creation"></a><span class="term">
1051      <code class="option">--ignore-thread-creation=&lt;yes|no&gt;
1052      [default: no]</code>
1053    </span>
1054</dt>
1055<dd>
1056<p>
1057        Controls whether all activities during thread creation should be
1058        ignored. By default enabled only on Solaris.
1059        Solaris provides higher throughput, parallelism and scalability than
1060        other operating systems, at the cost of more fine-grained locking
1061        activity. This means for example that when a thread is created under
1062        glibc, just one big lock is used for all thread setup. Solaris libc
1063        uses several fine-grained locks and the creator thread resumes its
1064        activities as soon as possible, leaving for example stack and TLS setup
1065        sequence to the created thread.
1066        This situation confuses Helgrind as it assumes there is some false
1067        ordering in place between creator and created thread; and therefore many
1068        types of race conditions in the application would not be reported.
1069        To prevent such false ordering, this command line option is set to
1070        <code class="computeroutput">yes</code> by default on Solaris.
1071        All activity (loads, stores, client requests) is therefore ignored
1072        during:</p>
1073<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1074<li class="listitem"><p>
1075            pthread_create() call in the creator thread
1076          </p></li>
1077<li class="listitem"><p>
1078            thread creation phase (stack and TLS setup) in the created thread
1079          </p></li>
1080</ul></div>
1081<p>
1082         Also new memory allocated during thread creation is untracked,
1083         that is race reporting is suppressed there. DRD does the same thing
1084         implicitly. This is necessary because Solaris libc caches many objects
1085         and reuses them for different threads and that confuses
1086         Helgrind.</p>
1087</dd>
1088</dl>
1089</div>
1090</div>
1091<div class="sect1">
1092<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1093<a name="hg-manual.monitor-commands"></a>7.7.�Helgrind Monitor Commands</h2></div></div></div>
1094<p>The Helgrind tool provides monitor commands handled by Valgrind's
1095built-in gdbserver (see <a class="xref" href="manual-core-adv.html#manual-core-adv.gdbserver-commandhandling" title="3.2.5.�Monitor command handling by the Valgrind gdbserver">Monitor command handling by the Valgrind gdbserver</a>).
1096</p>
1097<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1098<li class="listitem">
1099<p><code class="varname">info locks [lock_addr]</code> shows the list of locks
1100    and their status. If  <code class="varname">lock_addr</code> is given, only shows
1101    the lock located at this address. </p>
1102<p>
1103    In the following example, helgrind knows about one lock.  This
1104    lock is located at the guest address <code class="varname">ga
1105    0x8049a20</code>.  The lock kind is <code class="varname">rdwr</code>
1106    indicating a reader-writer lock.  Other possible lock kinds
1107    are <code class="varname">nonRec</code> (simple mutex, non recursive)
1108    and <code class="varname">mbRec</code> (simple mutex, possibly recursive).
1109    The lock kind is then followed by the list of threads helding the
1110    lock.  In the below example, <code class="varname">R1:thread #6 tid 3</code>
1111    indicates that the helgrind thread #6 has acquired (once, as the
1112    counter following the letter R is one) the lock in read mode. The
1113    helgrind thread nr is incremented for each started thread.  The
1114    presence of 'tid 3' indicates that the thread #6 is has not exited
1115    yet and is the valgrind tid 3. If a thread has terminated, then
1116    this is indicated with 'tid (exited)'.
1117    </p>
1118<pre class="programlisting">
1119(gdb) monitor info locks
1120Lock ga 0x8049a20 {
1121   kind   rdwr
1122 { R1:thread #6 tid 3 }
1123}
1124(gdb)
1125</pre>
1126<p> If you give the option <code class="varname">--read-var-info=yes</code>,
1127    then more information will be provided about the lock location, such as
1128    the global variable or the heap block that contains the lock:
1129    </p>
1130<pre class="programlisting">
1131Lock ga 0x8049a20 {
1132 Location 0x8049a20 is 0 bytes inside global var "s_rwlock"
1133 declared at rwlock_race.c:17
1134   kind   rdwr
1135 { R1:thread #3 tid 3 }
1136}
1137</pre>
1138</li>
1139<li class="listitem">
1140<p><code class="varname">accesshistory  &lt;addr&gt; [&lt;len&gt;]</code>
1141    shows the  access history recorded for &lt;len&gt; (default 1) bytes
1142    starting at &lt;addr&gt;. For each recorded access that overlaps
1143    with the given range, <code class="varname">accesshistory</code> shows the operation
1144    type (read or write), the address and size read or written, the helgrind
1145    thread nr/valgrind tid number that did the operation and the locks held
1146    by the thread at the time of the operation.
1147    The oldest access is shown first, the most recent access is shown last.
1148    </p>
1149<p>
1150    In the following example, we see first a recorded write of 4 bytes by
1151    thread #7 that has modified the given 2 bytes range.
1152    The second recorded write is the most recent recorded write : thread #9
1153    modified the same 2 bytes as part of a 4 bytes write operation.
1154    The list of locks held by each thread at the time of the write operation
1155    are also shown.
1156    </p>
1157<pre class="programlisting">
1158(gdb) monitor accesshistory 0x8049D8A 2
1159write of size 4 at 0x8049D88 by thread #7 tid 3
1160==6319== Locks held: 2, at address 0x8049D8C (and 1 that can't be shown)
1161==6319==    at 0x804865F: child_fn1 (locked_vs_unlocked2.c:29)
1162==6319==    by 0x400AE61: mythread_wrapper (hg_intercepts.c:234)
1163==6319==    by 0x39B924: start_thread (pthread_create.c:297)
1164==6319==    by 0x2F107D: clone (clone.S:130)
1165
1166write of size 4 at 0x8049D88 by thread #9 tid 2
1167==6319== Locks held: 2, at addresses 0x8049DA4 0x8049DD4
1168==6319==    at 0x804877B: child_fn2 (locked_vs_unlocked2.c:45)
1169==6319==    by 0x400AE61: mythread_wrapper (hg_intercepts.c:234)
1170==6319==    by 0x39B924: start_thread (pthread_create.c:297)
1171==6319==    by 0x2F107D: clone (clone.S:130)
1172
1173</pre>
1174</li>
1175</ul></div>
1176</div>
1177<div class="sect1">
1178<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1179<a name="hg-manual.client-requests"></a>7.8.�Helgrind Client Requests</h2></div></div></div>
1180<p>The following client requests are defined in
1181<code class="filename">helgrind.h</code>.  See that file for exact details of their
1182arguments.</p>
1183<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1184<li class="listitem">
1185<p><code class="function">VALGRIND_HG_CLEAN_MEMORY</code></p>
1186<p>This makes Helgrind forget everything it knows about a
1187    specified memory range.  This is particularly useful for memory
1188    allocators that wish to recycle memory.</p>
1189</li>
1190<li class="listitem"><p><code class="function">ANNOTATE_HAPPENS_BEFORE</code></p></li>
1191<li class="listitem"><p><code class="function">ANNOTATE_HAPPENS_AFTER</code></p></li>
1192<li class="listitem"><p><code class="function">ANNOTATE_NEW_MEMORY</code></p></li>
1193<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_CREATE</code></p></li>
1194<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_DESTROY</code></p></li>
1195<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_ACQUIRED</code></p></li>
1196<li class="listitem">
1197<p><code class="function">ANNOTATE_RWLOCK_RELEASED</code></p>
1198<p>These are used to describe to Helgrind, the behaviour of
1199    custom (non-POSIX) synchronisation primitives, which it otherwise
1200    has no way to understand.  See comments
1201    in <code class="filename">helgrind.h</code> for further
1202    documentation.</p>
1203</li>
1204</ul></div>
1205</div>
1206<div class="sect1">
1207<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1208<a name="hg-manual.todolist"></a>7.9.�A To-Do List for Helgrind</h2></div></div></div>
1209<p>The following is a list of loose ends which should be tidied up
1210some time.</p>
1211<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1212<li class="listitem"><p>For lock order errors, print the complete lock
1213    cycle, rather than only doing for size-2 cycles as at
1214    present.</p></li>
1215<li class="listitem"><p>The conflicting access mechanism sometimes
1216    mysteriously fails to show the conflicting access' stack, even
1217    when provided with unbounded storage for conflicting access info.
1218    This should be investigated.</p></li>
1219<li class="listitem"><p>Document races caused by GCC's thread-unsafe code
1220    generation for speculative stores.  In the interim see
1221    <code class="computeroutput">http://gcc.gnu.org/ml/gcc/2007-10/msg00266.html
1222    </code>
1223    and <code class="computeroutput">http://lkml.org/lkml/2007/10/24/673</code>.
1224    </p></li>
1225<li class="listitem"><p>Don't update the lock-order graph, and don't check
1226    for errors, when a "try"-style lock operation happens (e.g.
1227    <code class="function">pthread_mutex_trylock</code>).  Such calls do not add any real
1228    restrictions to the locking order, since they can always fail to
1229    acquire the lock, resulting in the caller going off and doing Plan
1230    B (presumably it will have a Plan B).  Doing such checks could
1231    generate false lock-order errors and confuse users.</p></li>
1232<li class="listitem"><p> Performance can be very poor.  Slowdowns on the
1233    order of 100:1 are not unusual.  There is limited scope for
1234    performance improvements.
1235    </p></li>
1236</ul></div>
1237</div>
1238</div>
1239<div>
1240<br><table class="nav" width="100%" cellspacing="3" cellpadding="2" border="0" summary="Navigation footer">
1241<tr>
1242<td rowspan="2" width="40%" align="left">
1243<a accesskey="p" href="cl-manual.html">&lt;&lt;�6.�Callgrind: a call-graph generating cache and branch prediction profiler</a>�</td>
1244<td width="20%" align="center"><a accesskey="u" href="manual.html">Up</a></td>
1245<td rowspan="2" width="40%" align="right">�<a accesskey="n" href="drd-manual.html">8.�DRD: a thread error detector�&gt;&gt;</a>
1246</td>
1247</tr>
1248<tr><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td></tr>
1249</table>
1250</div>
1251</body>
1252</html>
1253