README.md
1# benchmark
2[](https://travis-ci.org/google/benchmark)
3[](https://ci.appveyor.com/project/google/benchmark/branch/master)
4[](https://coveralls.io/r/google/benchmark)
5
6A library to support the benchmarking of functions, similar to unit-tests.
7
8Discussion group: https://groups.google.com/d/forum/benchmark-discuss
9
10IRC channel: https://freenode.net #googlebenchmark
11
12[Known issues and common problems](#known-issues)
13
14## Example usage
15### Basic usage
16Define a function that executes the code to be measured.
17
18```c++
19static void BM_StringCreation(benchmark::State& state) {
20 while (state.KeepRunning())
21 std::string empty_string;
22}
23// Register the function as a benchmark
24BENCHMARK(BM_StringCreation);
25
26// Define another benchmark
27static void BM_StringCopy(benchmark::State& state) {
28 std::string x = "hello";
29 while (state.KeepRunning())
30 std::string copy(x);
31}
32BENCHMARK(BM_StringCopy);
33
34BENCHMARK_MAIN();
35```
36
37### Passing arguments
38Sometimes a family of benchmarks can be implemented with just one routine that
39takes an extra argument to specify which one of the family of benchmarks to
40run. For example, the following code defines a family of benchmarks for
41measuring the speed of `memcpy()` calls of different lengths:
42
43```c++
44static void BM_memcpy(benchmark::State& state) {
45 char* src = new char[state.range(0)];
46 char* dst = new char[state.range(0)];
47 memset(src, 'x', state.range(0));
48 while (state.KeepRunning())
49 memcpy(dst, src, state.range(0));
50 state.SetBytesProcessed(int64_t(state.iterations()) *
51 int64_t(state.range(0)));
52 delete[] src;
53 delete[] dst;
54}
55BENCHMARK(BM_memcpy)->Arg(8)->Arg(64)->Arg(512)->Arg(1<<10)->Arg(8<<10);
56```
57
58The preceding code is quite repetitive, and can be replaced with the following
59short-hand. The following invocation will pick a few appropriate arguments in
60the specified range and will generate a benchmark for each such argument.
61
62```c++
63BENCHMARK(BM_memcpy)->Range(8, 8<<10);
64```
65
66By default the arguments in the range are generated in multiples of eight and
67the command above selects [ 8, 64, 512, 4k, 8k ]. In the following code the
68range multiplier is changed to multiples of two.
69
70```c++
71BENCHMARK(BM_memcpy)->RangeMultiplier(2)->Range(8, 8<<10);
72```
73Now arguments generated are [ 8, 16, 32, 64, 128, 256, 512, 1024, 2k, 4k, 8k ].
74
75You might have a benchmark that depends on two or more inputs. For example, the
76following code defines a family of benchmarks for measuring the speed of set
77insertion.
78
79```c++
80static void BM_SetInsert(benchmark::State& state) {
81 while (state.KeepRunning()) {
82 state.PauseTiming();
83 std::set<int> data = ConstructRandomSet(state.range(0));
84 state.ResumeTiming();
85 for (int j = 0; j < state.range(1); ++j)
86 data.insert(RandomNumber());
87 }
88}
89BENCHMARK(BM_SetInsert)
90 ->Args({1<<10, 1})
91 ->Args({1<<10, 8})
92 ->Args({1<<10, 64})
93 ->Args({1<<10, 512})
94 ->Args({8<<10, 1})
95 ->Args({8<<10, 8})
96 ->Args({8<<10, 64})
97 ->Args({8<<10, 512});
98```
99
100The preceding code is quite repetitive, and can be replaced with the following
101short-hand. The following macro will pick a few appropriate arguments in the
102product of the two specified ranges and will generate a benchmark for each such
103pair.
104
105```c++
106BENCHMARK(BM_SetInsert)->Ranges({{1<<10, 8<<10}, {1, 512}});
107```
108
109For more complex patterns of inputs, passing a custom function to `Apply` allows
110programmatic specification of an arbitrary set of arguments on which to run the
111benchmark. The following example enumerates a dense range on one parameter,
112and a sparse range on the second.
113
114```c++
115static void CustomArguments(benchmark::internal::Benchmark* b) {
116 for (int i = 0; i <= 10; ++i)
117 for (int j = 32; j <= 1024*1024; j *= 8)
118 b->Args({i, j});
119}
120BENCHMARK(BM_SetInsert)->Apply(CustomArguments);
121```
122
123### Calculate asymptotic complexity (Big O)
124Asymptotic complexity might be calculated for a family of benchmarks. The
125following code will calculate the coefficient for the high-order term in the
126running time and the normalized root-mean square error of string comparison.
127
128```c++
129static void BM_StringCompare(benchmark::State& state) {
130 std::string s1(state.range(0), '-');
131 std::string s2(state.range(0), '-');
132 while (state.KeepRunning()) {
133 benchmark::DoNotOptimize(s1.compare(s2));
134 }
135 state.SetComplexityN(state.range(0));
136}
137BENCHMARK(BM_StringCompare)
138 ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oN);
139```
140
141As shown in the following invocation, asymptotic complexity might also be
142calculated automatically.
143
144```c++
145BENCHMARK(BM_StringCompare)
146 ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity();
147```
148
149The following code will specify asymptotic complexity with a lambda function,
150that might be used to customize high-order term calculation.
151
152```c++
153BENCHMARK(BM_StringCompare)->RangeMultiplier(2)
154 ->Range(1<<10, 1<<18)->Complexity([](int n)->double{return n; });
155```
156
157### Templated benchmarks
158Templated benchmarks work the same way: This example produces and consumes
159messages of size `sizeof(v)` `range_x` times. It also outputs throughput in the
160absence of multiprogramming.
161
162```c++
163template <class Q> int BM_Sequential(benchmark::State& state) {
164 Q q;
165 typename Q::value_type v;
166 while (state.KeepRunning()) {
167 for (int i = state.range(0); i--; )
168 q.push(v);
169 for (int e = state.range(0); e--; )
170 q.Wait(&v);
171 }
172 // actually messages, not bytes:
173 state.SetBytesProcessed(
174 static_cast<int64_t>(state.iterations())*state.range(0));
175}
176BENCHMARK_TEMPLATE(BM_Sequential, WaitQueue<int>)->Range(1<<0, 1<<10);
177```
178
179Three macros are provided for adding benchmark templates.
180
181```c++
182#if __cplusplus >= 201103L // C++11 and greater.
183#define BENCHMARK_TEMPLATE(func, ...) // Takes any number of parameters.
184#else // C++ < C++11
185#define BENCHMARK_TEMPLATE(func, arg1)
186#endif
187#define BENCHMARK_TEMPLATE1(func, arg1)
188#define BENCHMARK_TEMPLATE2(func, arg1, arg2)
189```
190
191## Passing arbitrary arguments to a benchmark
192In C++11 it is possible to define a benchmark that takes an arbitrary number
193of extra arguments. The `BENCHMARK_CAPTURE(func, test_case_name, ...args)`
194macro creates a benchmark that invokes `func` with the `benchmark::State` as
195the first argument followed by the specified `args...`.
196The `test_case_name` is appended to the name of the benchmark and
197should describe the values passed.
198
199```c++
200template <class ...ExtraArgs>`
201void BM_takes_args(benchmark::State& state, ExtraArgs&&... extra_args) {
202 [...]
203}
204// Registers a benchmark named "BM_takes_args/int_string_test` that passes
205// the specified values to `extra_args`.
206BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));
207```
208Note that elements of `...args` may refer to global variables. Users should
209avoid modifying global state inside of a benchmark.
210
211## Using RegisterBenchmark(name, fn, args...)
212
213The `RegisterBenchmark(name, func, args...)` function provides an alternative
214way to create and register benchmarks.
215`RegisterBenchmark(name, func, args...)` creates, registers, and returns a
216pointer to a new benchmark with the specified `name` that invokes
217`func(st, args...)` where `st` is a `benchmark::State` object.
218
219Unlike the `BENCHMARK` registration macros, which can only be used at the global
220scope, the `RegisterBenchmark` can be called anywhere. This allows for
221benchmark tests to be registered programmatically.
222
223Additionally `RegisterBenchmark` allows any callable object to be registered
224as a benchmark. Including capturing lambdas and function objects. This
225allows the creation
226
227For Example:
228```c++
229auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };
230
231int main(int argc, char** argv) {
232 for (auto& test_input : { /* ... */ })
233 benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
234 benchmark::Initialize(&argc, argv);
235 benchmark::RunSpecifiedBenchmarks();
236}
237```
238
239### Multithreaded benchmarks
240In a multithreaded test (benchmark invoked by multiple threads simultaneously),
241it is guaranteed that none of the threads will start until all have called
242`KeepRunning`, and all will have finished before KeepRunning returns false. As
243such, any global setup or teardown can be wrapped in a check against the thread
244index:
245
246```c++
247static void BM_MultiThreaded(benchmark::State& state) {
248 if (state.thread_index == 0) {
249 // Setup code here.
250 }
251 while (state.KeepRunning()) {
252 // Run the test as normal.
253 }
254 if (state.thread_index == 0) {
255 // Teardown code here.
256 }
257}
258BENCHMARK(BM_MultiThreaded)->Threads(2);
259```
260
261If the benchmarked code itself uses threads and you want to compare it to
262single-threaded code, you may want to use real-time ("wallclock") measurements
263for latency comparisons:
264
265```c++
266BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime();
267```
268
269Without `UseRealTime`, CPU time is used by default.
270
271
272## Manual timing
273For benchmarking something for which neither CPU time nor real-time are
274correct or accurate enough, completely manual timing is supported using
275the `UseManualTime` function.
276
277When `UseManualTime` is used, the benchmarked code must call
278`SetIterationTime` once per iteration of the `KeepRunning` loop to
279report the manually measured time.
280
281An example use case for this is benchmarking GPU execution (e.g. OpenCL
282or CUDA kernels, OpenGL or Vulkan or Direct3D draw calls), which cannot
283be accurately measured using CPU time or real-time. Instead, they can be
284measured accurately using a dedicated API, and these measurement results
285can be reported back with `SetIterationTime`.
286
287```c++
288static void BM_ManualTiming(benchmark::State& state) {
289 int microseconds = state.range(0);
290 std::chrono::duration<double, std::micro> sleep_duration {
291 static_cast<double>(microseconds)
292 };
293
294 while (state.KeepRunning()) {
295 auto start = std::chrono::high_resolution_clock::now();
296 // Simulate some useful workload with a sleep
297 std::this_thread::sleep_for(sleep_duration);
298 auto end = std::chrono::high_resolution_clock::now();
299
300 auto elapsed_seconds =
301 std::chrono::duration_cast<std::chrono::duration<double>>(
302 end - start);
303
304 state.SetIterationTime(elapsed_seconds.count());
305 }
306}
307BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime();
308```
309
310### Preventing optimisation
311To prevent a value or expression from being optimized away by the compiler
312the `benchmark::DoNotOptimize(...)` and `benchmark::ClobberMemory()`
313functions can be used.
314
315```c++
316static void BM_test(benchmark::State& state) {
317 while (state.KeepRunning()) {
318 int x = 0;
319 for (int i=0; i < 64; ++i) {
320 benchmark::DoNotOptimize(x += i);
321 }
322 }
323}
324```
325
326`DoNotOptimize(<expr>)` forces the *result* of `<expr>` to be stored in either
327memory or a register. For GNU based compilers it acts as read/write barrier
328for global memory. More specifically it forces the compiler to flush pending
329writes to memory and reload any other values as necessary.
330
331Note that `DoNotOptimize(<expr>)` does not prevent optimizations on `<expr>`
332in any way. `<expr>` may even be removed entirely when the result is already
333known. For example:
334
335```c++
336 /* Example 1: `<expr>` is removed entirely. */
337 int foo(int x) { return x + 42; }
338 while (...) DoNotOptimize(foo(0)); // Optimized to DoNotOptimize(42);
339
340 /* Example 2: Result of '<expr>' is only reused */
341 int bar(int) __attribute__((const));
342 while (...) DoNotOptimize(bar(0)); // Optimized to:
343 // int __result__ = bar(0);
344 // while (...) DoNotOptimize(__result__);
345```
346
347The second tool for preventing optimizations is `ClobberMemory()`. In essence
348`ClobberMemory()` forces the compiler to perform all pending writes to global
349memory. Memory managed by block scope objects must be "escaped" using
350`DoNotOptimize(...)` before it can be clobbered. In the below example
351`ClobberMemory()` prevents the call to `v.push_back(42)` from being optimized
352away.
353
354```c++
355static void BM_vector_push_back(benchmark::State& state) {
356 while (state.KeepRunning()) {
357 std::vector<int> v;
358 v.reserve(1);
359 benchmark::DoNotOptimize(v.data()); // Allow v.data() to be clobbered.
360 v.push_back(42);
361 benchmark::ClobberMemory(); // Force 42 to be written to memory.
362 }
363}
364```
365
366Note that `ClobberMemory()` is only available for GNU based compilers.
367
368### Set time unit manually
369If a benchmark runs a few milliseconds it may be hard to visually compare the
370measured times, since the output data is given in nanoseconds per default. In
371order to manually set the time unit, you can specify it manually:
372
373```c++
374BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
375```
376
377## Controlling number of iterations
378In all cases, the number of iterations for which the benchmark is run is
379governed by the amount of time the benchmark takes. Concretely, the number of
380iterations is at least one, not more than 1e9, until CPU time is greater than
381the minimum time, or the wallclock time is 5x minimum time. The minimum time is
382set as a flag `--benchmark_min_time` or per benchmark by calling `MinTime` on
383the registered benchmark object.
384
385## Reporting the mean and standard devation by repeated benchmarks
386By default each benchmark is run once and that single result is reported.
387However benchmarks are often noisy and a single result may not be representative
388of the overall behavior. For this reason it's possible to repeatedly rerun the
389benchmark.
390
391The number of runs of each benchmark is specified globally by the
392`--benchmark_repetitions` flag or on a per benchmark basis by calling
393`Repetitions` on the registered benchmark object. When a benchmark is run
394more than once the mean and standard deviation of the runs will be reported.
395
396Additionally the `--benchmark_report_aggregates_only={true|false}` flag or
397`ReportAggregatesOnly(bool)` function can be used to change how repeated tests
398are reported. By default the result of each repeated run is reported. When this
399option is 'true' only the mean and standard deviation of the runs is reported.
400Calling `ReportAggregatesOnly(bool)` on a registered benchmark object overrides
401the value of the flag for that benchmark.
402
403## Fixtures
404Fixture tests are created by
405first defining a type that derives from ::benchmark::Fixture and then
406creating/registering the tests using the following macros:
407
408* `BENCHMARK_F(ClassName, Method)`
409* `BENCHMARK_DEFINE_F(ClassName, Method)`
410* `BENCHMARK_REGISTER_F(ClassName, Method)`
411
412For Example:
413
414```c++
415class MyFixture : public benchmark::Fixture {};
416
417BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) {
418 while (st.KeepRunning()) {
419 ...
420 }
421}
422
423BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
424 while (st.KeepRunning()) {
425 ...
426 }
427}
428/* BarTest is NOT registered */
429BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
430/* BarTest is now registered */
431```
432
433## Exiting Benchmarks in Error
434
435When errors caused by external influences, such as file I/O and network
436communication, occur within a benchmark the
437`State::SkipWithError(const char* msg)` function can be used to skip that run
438of benchmark and report the error. Note that only future iterations of the
439`KeepRunning()` are skipped. Users may explicitly return to exit the
440benchmark immediately.
441
442The `SkipWithError(...)` function may be used at any point within the benchmark,
443including before and after the `KeepRunning()` loop.
444
445For example:
446
447```c++
448static void BM_test(benchmark::State& state) {
449 auto resource = GetResource();
450 if (!resource.good()) {
451 state.SkipWithError("Resource is not good!");
452 // KeepRunning() loop will not be entered.
453 }
454 while (state.KeepRunning()) {
455 auto data = resource.read_data();
456 if (!resource.good()) {
457 state.SkipWithError("Failed to read data!");
458 break; // Needed to skip the rest of the iteration.
459 }
460 do_stuff(data);
461 }
462}
463```
464
465## Running a subset of the benchmarks
466
467The `--benchmark_filter=<regex>` option can be used to only run the benchmarks
468which match the specified `<regex>`. For example:
469
470```bash
471$ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
472Run on (1 X 2300 MHz CPU )
4732016-06-25 19:34:24
474Benchmark Time CPU Iterations
475----------------------------------------------------
476BM_memcpy/32 11 ns 11 ns 79545455
477BM_memcpy/32k 2181 ns 2185 ns 324074
478BM_memcpy/32 12 ns 12 ns 54687500
479BM_memcpy/32k 1834 ns 1837 ns 357143
480```
481
482
483## Output Formats
484The library supports multiple output formats. Use the
485`--benchmark_format=<console|json|csv>` flag to set the format type. `console`
486is the default format.
487
488The Console format is intended to be a human readable format. By default
489the format generates color output. Context is output on stderr and the
490tabular data on stdout. Example tabular output looks like:
491```
492Benchmark Time(ns) CPU(ns) Iterations
493----------------------------------------------------------------------
494BM_SetInsert/1024/1 28928 29349 23853 133.097kB/s 33.2742k items/s
495BM_SetInsert/1024/8 32065 32913 21375 949.487kB/s 237.372k items/s
496BM_SetInsert/1024/10 33157 33648 21431 1.13369MB/s 290.225k items/s
497```
498
499The JSON format outputs human readable json split into two top level attributes.
500The `context` attribute contains information about the run in general, including
501information about the CPU and the date.
502The `benchmarks` attribute contains a list of ever benchmark run. Example json
503output looks like:
504``` json
505{
506 "context": {
507 "date": "2015/03/17-18:40:25",
508 "num_cpus": 40,
509 "mhz_per_cpu": 2801,
510 "cpu_scaling_enabled": false,
511 "build_type": "debug"
512 },
513 "benchmarks": [
514 {
515 "name": "BM_SetInsert/1024/1",
516 "iterations": 94877,
517 "real_time": 29275,
518 "cpu_time": 29836,
519 "bytes_per_second": 134066,
520 "items_per_second": 33516
521 },
522 {
523 "name": "BM_SetInsert/1024/8",
524 "iterations": 21609,
525 "real_time": 32317,
526 "cpu_time": 32429,
527 "bytes_per_second": 986770,
528 "items_per_second": 246693
529 },
530 {
531 "name": "BM_SetInsert/1024/10",
532 "iterations": 21393,
533 "real_time": 32724,
534 "cpu_time": 33355,
535 "bytes_per_second": 1199226,
536 "items_per_second": 299807
537 }
538 ]
539}
540```
541
542The CSV format outputs comma-separated values. The `context` is output on stderr
543and the CSV itself on stdout. Example CSV output looks like:
544```
545name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
546"BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
547"BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
548"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,
549```
550
551## Output Files
552The library supports writing the output of the benchmark to a file specified
553by `--benchmark_out=<filename>`. The format of the output can be specified
554using `--benchmark_out_format={json|console|csv}`. Specifying
555`--benchmark_out` does not suppress the console output.
556
557## Debug vs Release
558By default, benchmark builds as a debug library. You will see a warning in the output when this is the case. To build it as a release library instead, use:
559
560```
561cmake -DCMAKE_BUILD_TYPE=Release
562```
563
564To enable link-time optimisation, use
565
566```
567cmake -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_LTO=true
568```
569
570## Linking against the library
571When using gcc, it is necessary to link against pthread to avoid runtime exceptions.
572This is due to how gcc implements std::thread.
573See [issue #67](https://github.com/google/benchmark/issues/67) for more details.
574
575## Compiler Support
576
577Google Benchmark uses C++11 when building the library. As such we require
578a modern C++ toolchain, both compiler and standard library.
579
580The following minimum versions are strongly recommended build the library:
581
582* GCC 4.8
583* Clang 3.4
584* Visual Studio 2013
585
586Anything older *may* work.
587
588Note: Using the library and its headers in C++03 is supported. C++11 is only
589required to build the library.
590
591# Known Issues
592
593### Windows
594
595* Users must manually link `shlwapi.lib`. Failure to do so may result
596in unresolved symbols.
597
598
){kind=link}
){kind=link}