1# SafetyNet - Performance regression detection for PDFium
2
3[TOC]
4
5This document explains how to use SafetyNet to detect performance regressions
6in PDFium.
7
8## Comparing performance of two versions of PDFium
9
10safetynet_compare.py is a script that compares the performance between two
11versions of pdfium. This can be used to verify if a given change has caused
12or will cause any positive or negative changes in performance for a set of test
13cases.
14
15The supported profilers are exclusive to Linux, so for now this can only be run
16on Linux.
17
18An illustrative example is below, comparing the local code version to an older
19version. Positive % changes mean an increase in time/instructions to run the
20test - a regression, while negative % changes mean a decrease in
21time/instructions, therefore an improvement.
22
23```
24$ testing/tools/safetynet_compare.py ~/test_pdfs --branch-before beef5e4
25================================================================================
26   % Change      Time after  Test case
27--------------------------------------------------------------------------------
28   -0.1980%  45,703,820,326  ~/test_pdfs/PDF Reference 1-7.pdf
29   -0.5678%      42,038,814  ~/test_pdfs/Page 24 - PDF Reference 1-7.pdf
30   +0.2666%  10,983,158,809  ~/test_pdfs/Rival.pdf
31   +0.0447%  10,413,890,748  ~/test_pdfs/dynamic.pdf
32   -7.7228%      26,161,171  ~/test_pdfs/encrypted1234.pdf
33   -0.2763%     102,084,398  ~/test_pdfs/ghost.pdf
34   -3.7005%  10,800,642,262  ~/test_pdfs/musician.pdf
35   -0.2266%  45,691,618,789  ~/test_pdfs/no_metadata.pdf
36   +1.4440%  38,442,606,162  ~/test_pdfs/test7.pdf
37   +0.0335%       9,286,083  ~/test_pdfs/testbulletpoint.pdf
38================================================================================
39Test cases run: 10
40Failed to measure: 0
41Regressions: 0
42Improvements: 2
43```
44
45### Usage
46
47Run the safetynet_compare.py script in testing/tools to perform a comparison.
48Pass one or more paths with test cases - each path can be either a .pdf file or
49a directory containing .pdf files. Other files in those directories are
50ignored.
51
52The following comparison modes are supported:
53
541. Compare uncommitted changes against clean branch:
55```shell
56$ testing/tools/safetynet_compare.py path/to/pdfs
57```
58
592. Compare current branch with another branch or commit:
60```shell
61$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before another_branch
62$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before 1a3c5e7
63```
64
653. Compare two other branches or commits:
66```shell
67$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before yet_another_branch
68$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after 1a3c5e7 --branch-before 0b2d4f6
69$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before 0b2d4f6
70```
71
724. Compare two build flag configurations:
73```shell
74$ gn args out/BuildConfig1
75$ gn args out/BuildConfig2
76$ testing/tools/safetynet_compare.py path/to/pdfs --build-dir out/BuildConfig2 --build-dir-before out/BuildConfig1
77```
78
79safetynet_compare.py takes care of checking out the appropriate branch, building
80it, running the test cases and comparing results.
81
82### Profilers
83
84safetynet_compare.py uses callgrind as a profiler by default. Use --profiler
85to specify another one. The supported ones are:
86
87#### perfstat
88
89Only works on Linux.
90Make sure you have perf by typing in the terminal:
91```shell
92$ perf
93```
94
95This is a fast profiler, but uses sampling so it's slightly inaccurate.
96Expect variations of up to 1%, which is below the cutoff to consider a
97change significant.
98
99Use this when running over large test sets to get good enough results.
100
101#### callgrind
102
103Only works on Linux.
104Make sure valgrind is installed:
105```shell
106$ valgrind
107```
108
109This is a slow and accurate profiler. Expect variations of around 100
110instructions. However, this takes about 50 times longer to run than perf stat.
111
112Use this when looking for small variations (< 1%).
113
114One advantage is that callgrind can generate `callgrind.out` files (by passing
115--output-dir to safetynet_compare.py), which contain profiling information that
116can be analyzed to find the cause of a regression. KCachegrind is a good
117visualizer for these files.
118
119### Common Options
120
121Arguments commonly passed to safetynet_compare.py.
122
123* --profiler: described above.
124* --build-dir: this specified the build config with a relative path from the
125pdfium src directory to the build directory. Defaults to out/Release.
126* --output-dir: where to place the profiling output files. These are
127callgrind.out.[test_case] files for callgrind, perfstat does not produce them.
128By default they are not written.
129* --case-order: sort test case results according to this metric. Can be "after",
130"before", "ratio" and "rating". If not specified, sort by path.
131* --this-repo: use the repository where the script is instead of checking out a
132temporary one. This is faster and does not require downloads. Although it
133restores the state of the local repo, if the script is killed or crashes the
134uncommitted changes can remain stashed and you may be on another branch.
135
136### Other Options
137
138Most of the time these don't need to be used.
139
140* --build-dir-before: if comparing different build dirs (say, to test what a
141flag flip does), specify the build dir for the “before” branch here and the
142build dir for the “after” branch with --build-dir.
143* --interesting-section: only the interesting section should be measured instead
144of all the execution of the test harness. This only works in debug, since in
145release the delimiters are stripped out. This does not work to compare branches
146that don’t have the callgrind delimiters, as it would otherwise be unfair to
147compare a whole run vs the interesting section of another run.
148* --machine-readable: output a json with the results that is easier to read by
149code.
150* --num-workers: how many workers to use to parallelize test case runs. Defaults
151to # of CPUs in the machine.
152* --threshold-significant: highlight differences that exceed this value.
153Defaults to 0.02.
154* --tmp-dir: directory in which temporary repos will be cloned and downloads
155will be cached, if --this-repo is not enabled. Defaults to /tmp.
156
157## Setup a nightly job
158
159TODO: Complete with safetynet_job.py setup and usage.
160