1test-suite Guide 2================ 3 4Quickstart 5---------- 6 71. The lit test runner is required to run the tests. You can either use one 8 from an LLVM build: 9 10 ```bash 11 % <path to llvm build>/bin/llvm-lit --version 12 lit 0.8.0dev 13 ``` 14 15 An alternative is installing it as a python package in a python virtual 16 environment: 17 18 ```bash 19 % mkdir venv 20 % virtualenv venv 21 % . venv/bin/activate 22 % pip install svn+https://llvm.org/svn/llvm-project/llvm/trunk/utils/lit 23 % lit --version 24 lit 0.8.0dev 25 ``` 26 272. Check out the `test-suite` module with: 28 29 ```bash 30 % git clone https://github.com/llvm/llvm-test-suite.git test-suite 31 ``` 32 333. Create a build directory and use CMake to configure the suite. Use the 34 `CMAKE_C_COMPILER` option to specify the compiler to test. Use a cache file 35 to choose a typical build configuration: 36 37 ```bash 38 % mkdir test-suite-build 39 % cd test-suite-build 40 % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \ 41 -C../test-suite/cmake/caches/O3.cmake \ 42 ../test-suite 43 ``` 44 454. Build the benchmarks: 46 47 ```text 48 % make 49 Scanning dependencies of target timeit-target 50 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o 51 [ 0%] Linking C executable timeit-target 52 ... 53 ``` 54 555. Run the tests with lit: 56 57 ```text 58 % llvm-lit -v -j 1 -o results.json . 59 -- Testing: 474 tests, 1 threads -- 60 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474) 61 ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS ********** 62 compile_time: 0.2192 63 exec_time: 0.0462 64 hash: "59620e187c6ac38b36382685ccd2b63b" 65 size: 83348 66 ********** 67 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474) 68 ... 69 ``` 70 716. Show and compare result files (optional): 72 73 ```bash 74 # Make sure pandas and scipy are installed. Prepend `sudo` if necessary. 75 % pip install pandas scipy 76 # Show a single result file: 77 % test-suite/utils/compare.py results.json 78 # Compare two result files: 79 % test-suite/utils/compare.py results_a.json results_b.json 80 ``` 81 82 83Structure 84--------- 85 86The test-suite contains benchmark and test programs. The programs come with 87reference outputs so that their correctness can be checked. The suite comes 88with tools to collect metrics such as benchmark runtime, compilation time and 89code size. 90 91The test-suite is divided into several directories: 92 93- `SingleSource/` 94 95 Contains test programs that are only a single source file in size. A 96 subdirectory may contain several programs. 97 98- `MultiSource/` 99 100 Contains subdirectories which entire programs with multiple source files. 101 Large benchmarks and whole applications go here. 102 103- `MicroBenchmarks/` 104 105 Programs using the [google-benchmark](https://github.com/google/benchmark) 106 library. The programs define functions that are run multiple times until the 107 measurement results are statistically significant. 108 109- `External/` 110 111 Contains descriptions and test data for code that cannot be directly 112 distributed with the test-suite. The most prominent members of this 113 directory are the SPEC CPU benchmark suites. 114 See [External Suites](#external-suites). 115 116- `Bitcode/` 117 118 These tests are mostly written in LLVM bitcode. 119 120- `CTMark/` 121 122 Contains symbolic links to other benchmarks forming a representative sample 123 for compilation performance measurements. 124 125### Benchmarks 126 127Every program can work as a correctness test. Some programs are unsuitable for 128performance measurements. Setting the `TEST_SUITE_BENCHMARKING_ONLY` CMake 129option to `ON` will disable them. 130 131 132Configuration 133------------- 134 135The test-suite has configuration options to customize building and running the 136benchmarks. CMake can print a list of them: 137 138```bash 139% cd test-suite-build 140# Print basic options: 141% cmake -LH 142# Print all options: 143% cmake -LAH 144``` 145 146### Common Configuration Options 147 148- `CMAKE_C_FLAGS` 149 150 Specify extra flags to be passed to C compiler invocations. The flags are 151 also passed to the C++ compiler and linker invocations. See 152 [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html) 153 154- `CMAKE_C_COMPILER` 155 156 Select the C compiler executable to be used. Note that the C++ compiler is 157 inferred automatically i.e. when specifying `path/to/clang` CMake will 158 automatically use `path/to/clang++` as the C++ compiler. See 159 [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html) 160 161- `CMAKE_BUILD_TYPE` 162 163 Select a build type like `OPTIMIZE` or `DEBUG` selecting a set of predefined 164 compiler flags. These flags are applied regardless of the `CMAKE_C_FLAGS` 165 option and may be changed by modifying `CMAKE_C_FLAGS_OPTIMIZE` etc. See 166 [https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html](https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html) 167 168- `TEST_SUITE_RUN_UNDER` 169 170 Prefix test invocations with the given tool. This is typically used to run 171 cross-compiled tests within a simulator tool. 172 173- `TEST_SUITE_BENCHMARKING_ONLY` 174 175 Disable tests that are unsuitable for performance measurements. The disabled 176 tests either run for a very short time or are dominated by I/O performance 177 making them unsuitable as compiler performance tests. 178 179- `TEST_SUITE_SUBDIRS` 180 181 Semicolon-separated list of directories to include. This can be used to only 182 build parts of the test-suite or to include external suites. This option 183 does not work reliably with deeper subdirectories as it skips intermediate 184 `CMakeLists.txt` files which may be required. 185 186- `TEST_SUITE_COLLECT_STATS` 187 188 Collect internal LLVM statistics. Appends `-save-stats=obj` when invoking the 189 compiler and makes the lit runner collect and merge the statistic files. 190 191- `TEST_SUITE_RUN_BENCHMARKS` 192 193 If this is set to `OFF` then lit will not actually run the tests but just 194 collect build statistics like compile time and code size. 195 196- `TEST_SUITE_USE_PERF` 197 198 Use the `perf` tool for time measurement instead of the `timeit` tool that 199 comes with the test-suite. The `perf` is usually available on linux systems. 200 201- `TEST_SUITE_SPEC2000_ROOT`, `TEST_SUITE_SPEC2006_ROOT`, `TEST_SUITE_SPEC2017_ROOT`, ... 202 203 Specify installation directories of external benchmark suites. You can find 204 more information about expected versions or usage in the README files in the 205 `External` directory (such as `External/SPEC/README`) 206 207### Common CMake Flags 208 209- `-GNinja` 210 211 Generate build files for the ninja build tool. 212 213- `-Ctest-suite/cmake/caches/<cachefile.cmake>` 214 215 Use a CMake cache. The test-suite comes with several CMake caches which 216 predefine common or tricky build configurations. 217 218 219Displaying and Analyzing Results 220-------------------------------- 221 222The `compare.py` script displays and compares result files. A result file is 223produced when invoking lit with the `-o filename.json` flag. 224 225Example usage: 226 227- Basic Usage: 228 229 ```text 230 % test-suite/utils/compare.py baseline.json 231 Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics! 232 Tests: 508 233 Metric: exec_time 234 235 Program baseline 236 237 INT2006/456.hmmer/456.hmmer 1222.90 238 INT2006/464.h264ref/464.h264ref 928.70 239 ... 240 baseline 241 count 506.000000 242 mean 20.563098 243 std 111.423325 244 min 0.003400 245 25% 0.011200 246 50% 0.339450 247 75% 4.067200 248 max 1222.896800 249 ``` 250 251- Show compile_time or text segment size metrics: 252 253 ```bash 254 % test-suite/utils/compare.py -m compile_time baseline.json 255 % test-suite/utils/compare.py -m size.__text baseline.json 256 ``` 257 258- Compare two result files and filter short running tests: 259 260 ```bash 261 % test-suite/utils/compare.py --filter-short baseline.json experiment.json 262 ... 263 Program baseline experiment diff 264 265 SingleSour.../Benchmarks/Linpack/linpack-pc 5.16 4.30 -16.5% 266 MultiSourc...erolling-dbl/LoopRerolling-dbl 7.01 7.86 12.2% 267 SingleSour...UnitTests/Vectorizer/gcc-loops 3.89 3.54 -9.0% 268 ... 269 ``` 270 271- Merge multiple baseline and experiment result files by taking the minimum 272 runtime each: 273 274 ```bash 275 % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json 276 ``` 277 278### Continuous Tracking with LNT 279 280LNT is a set of client and server tools for continuously monitoring 281performance. You can find more information at 282[https://llvm.org/docs/lnt](https://llvm.org/docs/lnt). The official LNT instance 283of the LLVM project is hosted at [http://lnt.llvm.org](http://lnt.llvm.org). 284 285 286External Suites 287--------------- 288 289External suites such as SPEC can be enabled by either 290 291- placing (or linking) them into the `test-suite/test-suite-externals/xxx` directory (example: `test-suite/test-suite-externals/speccpu2000`) 292- using a configuration option such as `-D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000` 293 294You can find further information in the respective README files such as 295`test-suite/External/SPEC/README`. 296 297For the SPEC benchmarks you can switch between the `test`, `train` and 298`ref` input datasets via the `TEST_SUITE_RUN_TYPE` configuration option. 299The `train` dataset is used by default. 300 301 302Custom Suites 303------------- 304 305You can build custom suites using the test-suite infrastructure. A custom suite 306has a `CMakeLists.txt` file at the top directory. The `CMakeLists.txt` will be 307picked up automatically if placed into a subdirectory of the test-suite or when 308setting the `TEST_SUITE_SUBDIRS` variable: 309 310```bash 311% cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite 312``` 313 314 315Profile Guided Optimization 316--------------------------- 317 318Profile guided optimization requires to compile and run twice. First the 319benchmark should be compiled with profile generation instrumentation enabled 320and setup for training data. The lit runner will merge the profile files 321using `llvm-profdata` so they can be used by the second compilation run. 322 323Example: 324```bash 325# Profile generation run: 326% cmake -DTEST_SUITE_PROFILE_GENERATE=ON \ 327 -DTEST_SUITE_RUN_TYPE=train \ 328 ../test-suite 329% make 330% llvm-lit . 331# Use the profile data for compilation and actual benchmark run: 332% cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \ 333 -DTEST_SUITE_PROFILE_USE=ON \ 334 -DTEST_SUITE_RUN_TYPE=ref \ 335 . 336% make 337% llvm-lit -o result.json . 338``` 339 340The `TEST_SUITE_RUN_TYPE` setting only affects the SPEC benchmark suites. 341 342 343Cross Compilation and External Devices 344-------------------------------------- 345 346### Compilation 347 348CMake allows to cross compile to a different target via toolchain files. More 349information can be found here: 350 351- [https://llvm.org/docs/lnt/tests.html#cross-compiling](https://llvm.org/docs/lnt/tests.html#cross-compiling) 352 353- [https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html) 354 355Cross compilation from macOS to iOS is possible with the 356`test-suite/cmake/caches/target-target-*-iphoneos-internal.cmake` CMake cache 357files; this requires an internal iOS SDK. 358 359### Running 360 361There are two ways to run the tests in a cross compilation setting: 362 363- Via SSH connection to an external device: The `TEST_SUITE_REMOTE_HOST` option 364 should be set to the SSH hostname. The executables and data files need to be 365 transferred to the device after compilation. This is typically done via the 366 `rsync` make target. After this, the lit runner can be used on the host 367 machine. It will prefix the benchmark and verification command lines with an 368 `ssh` command. 369 370 Example: 371 372 ```bash 373 % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \ 374 -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \ 375 -D TEST_SUITE_REMOTE_HOST=mydevice \ 376 ../test-suite 377 % ninja 378 % ninja rsync 379 % llvm-lit -j1 -o result.json . 380 ``` 381 382- You can specify a simulator for the target machine with the 383 `TEST_SUITE_RUN_UNDER` setting. The lit runner will prefix all benchmark 384 invocations with it. 385 386 387Running the test-suite via LNT 388------------------------------ 389 390The LNT tool can run the test-suite. Use this when submitting test results to 391an LNT instance. See 392[https://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite](https://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite) 393for details. 394 395Running the test-suite via Makefiles (deprecated) 396------------------------------------------------- 397 398**Note**: The test-suite comes with a set of Makefiles that are considered 399deprecated. They do not support newer testing modes like `Bitcode` or 400`Microbenchmarks` and are harder to use. 401 402Old documentation is available in the 403[test-suite Makefile Guide](TestSuiteMakefileGuide). 404