1==================== 2XRay Instrumentation 3==================== 4 5:Version: 1 as of 2016-11-08 6 7.. contents:: 8 :local: 9 10 11Introduction 12============ 13 14XRay is a function call tracing system which combines compiler-inserted 15instrumentation points and a runtime library that can dynamically enable and 16disable the instrumentation. 17 18More high level information about XRay can be found in the `XRay whitepaper`_. 19 20This document describes how to use XRay as implemented in LLVM. 21 22XRay in LLVM 23============ 24 25XRay consists of three main parts: 26 27- Compiler-inserted instrumentation points. 28- A runtime library for enabling/disabling tracing at runtime. 29- A suite of tools for analysing the traces. 30 31 **NOTE:** As of July 25, 2018 , XRay is only available for the following 32 architectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le, 33 mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 and 34 OpenBSD: x86_64. 35 36The compiler-inserted instrumentation points come in the form of nop-sleds in 37the final generated binary, and an ELF section named ``xray_instr_map`` which 38contains entries pointing to these instrumentation points. The runtime library 39relies on being able to access the entries of the ``xray_instr_map``, and 40overwrite the instrumentation points at runtime. 41 42Using XRay 43========== 44 45You can use XRay in a couple of ways: 46 47- Instrumenting your C/C++/Objective-C/Objective-C++ application. 48- Generating LLVM IR with the correct function attributes. 49 50The rest of this section covers these main ways and later on how to customise 51what XRay does in an XRay-instrumented binary. 52 53Instrumenting your C/C++/Objective-C Application 54------------------------------------------------ 55 56The easiest way of getting XRay instrumentation for your application is by 57enabling the ``-fxray-instrument`` flag in your clang invocation. 58 59For example: 60 61:: 62 63 clang -fxray-instrument ... 64 65By default, functions that have at least 200 instructions (or contain a loop) will 66get XRay instrumentation points. You can tweak that number through the 67``-fxray-instruction-threshold=`` flag: 68 69:: 70 71 clang -fxray-instrument -fxray-instruction-threshold=1 ... 72 73The loop detection can be disabled with ``-fxray-ignore-loops`` to use only the 74instruction threshold. You can also specifically instrument functions in your 75binary to either always or never be instrumented using source-level attributes. 76You can do it using the GCC-style attributes or C++11-style attributes. 77 78.. code-block:: c++ 79 80 [[clang::xray_always_instrument]] void always_instrumented(); 81 82 [[clang::xray_never_instrument]] void never_instrumented(); 83 84 void alt_always_instrumented() __attribute__((xray_always_instrument)); 85 86 void alt_never_instrumented() __attribute__((xray_never_instrument)); 87 88When linking a binary, you can either manually link in the `XRay Runtime 89Library`_ or use ``clang`` to link it in automatically with the 90``-fxray-instrument`` flag. Alternatively, you can statically link-in the XRay 91runtime library from compiler-rt -- those archive files will take the name of 92`libclang_rt.xray-{arch}` where `{arch}` is the mnemonic supported by clang 93(x86_64, arm7, etc.). 94 95LLVM Function Attribute 96----------------------- 97 98If you're using LLVM IR directly, you can add the ``function-instrument`` 99string attribute to your functions, to get the similar effect that the 100C/C++/Objective-C source-level attributes would get: 101 102.. code-block:: llvm 103 104 define i32 @always_instrument() uwtable "function-instrument"="xray-always" { 105 ; ... 106 } 107 108 define i32 @never_instrument() uwtable "function-instrument"="xray-never" { 109 ; ... 110 } 111 112You can also set the ``xray-instruction-threshold`` attribute and provide a 113numeric string value for how many instructions should be in the function before 114it gets instrumented. 115 116.. code-block:: llvm 117 118 define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" { 119 ; ... 120 } 121 122Special Case File 123----------------- 124 125Attributes can be imbued through the use of special case files instead of 126adding them to the original source files. You can use this to mark certain 127functions and classes to be never, always, or instrumented with first-argument 128logging from a file. The file's format is described below: 129 130.. code-block:: bash 131 132 # Comments are supported 133 [always] 134 fun:always_instrument 135 fun:log_arg1=arg1 # Log the first argument for the function 136 137 [never] 138 fun:never_instrument 139 140These files can be provided through the ``-fxray-attr-list=`` flag to clang. 141You may have multiple files loaded through multiple instances of the flag. 142 143XRay Runtime Library 144-------------------- 145 146The XRay Runtime Library is part of the compiler-rt project, which implements 147the runtime components that perform the patching and unpatching of inserted 148instrumentation points. When you use ``clang`` to link your binaries and the 149``-fxray-instrument`` flag, it will automatically link in the XRay runtime. 150 151The default implementation of the XRay runtime will enable XRay instrumentation 152before ``main`` starts, which works for applications that have a short 153lifetime. This implementation also records all function entry and exit events 154which may result in a lot of records in the resulting trace. 155 156Also by default the filename of the XRay trace is ``xray-log.XXXXXX`` where the 157``XXXXXX`` part is randomly generated. 158 159These options can be controlled through the ``XRAY_OPTIONS`` environment 160variable, where we list down the options and their defaults below. 161 162+-------------------+-----------------+---------------+------------------------+ 163| Option | Type | Default | Description | 164+===================+=================+===============+========================+ 165| patch_premain | ``bool`` | ``false`` | Whether to patch | 166| | | | instrumentation points | 167| | | | before main. | 168+-------------------+-----------------+---------------+------------------------+ 169| xray_mode | ``const char*`` | ``""`` | Default mode to | 170| | | | install and initialize | 171| | | | before ``main``. | 172+-------------------+-----------------+---------------+------------------------+ 173| xray_logfile_base | ``const char*`` | ``xray-log.`` | Filename base for the | 174| | | | XRay logfile. | 175+-------------------+-----------------+---------------+------------------------+ 176| verbosity | ``int`` | ``0`` | Runtime verbosity | 177| | | | level. | 178+-------------------+-----------------+---------------+------------------------+ 179 180 181If you choose to not use the default logging implementation that comes with the 182XRay runtime and/or control when/how the XRay instrumentation runs, you may use 183the XRay APIs directly for doing so. To do this, you'll need to include the 184``xray_log_interface.h`` from the compiler-rt ``xray`` directory. The important API 185functions we list below: 186 187- ``__xray_log_register_mode(...)``: Register a logging implementation against 188 a string Mode identifier. The implementation is an instance of 189 ``XRayLogImpl`` defined in ``xray/xray_log_interface.h``. 190- ``__xray_log_select_mode(...)``: Select the mode to install, associated with 191 a string Mode identifier. Only implementations registered with 192 ``__xray_log_register_mode(...)`` can be chosen with this function. 193- ``__xray_log_init_mode(...)``: This function allows for initializing and 194 re-initializing an installed logging implementation. See 195 ``xray/xray_log_interface.h`` for details, part of the XRay compiler-rt 196 installation. 197 198Once a logging implementation has been initialized, it can be "stopped" by 199finalizing the implementation through the ``__xray_log_finalize()`` function. 200The finalization routine is the opposite of the initialization. When finalized, 201an implementation's data can be cleared out through the 202``__xray_log_flushLog()`` function. For implementations that support in-memory 203processing, these should register an iterator function to provide access to the 204data via the ``__xray_log_set_buffer_iterator(...)`` which allows code calling 205the ``__xray_log_process_buffers(...)`` function to deal with the data in 206memory. 207 208All of this is better explained in the ``xray/xray_log_interface.h`` header. 209 210Basic Mode 211---------- 212 213XRay supports a basic logging mode which will trace the application's 214execution, and periodically append to a single log. This mode can be 215installed/enabled by setting ``xray_mode=xray-basic`` in the ``XRAY_OPTIONS`` 216environment variable. Combined with ``patch_premain=true`` this can allow for 217tracing applications from start to end. 218 219Like all the other modes installed through ``__xray_log_select_mode(...)``, the 220implementation can be configured through the ``__xray_log_init_mode(...)`` 221function, providing the mode string and the flag options. Basic-mode specific 222defaults can be provided in the ``XRAY_BASIC_OPTIONS`` environment variable. 223 224Flight Data Recorder Mode 225------------------------- 226 227XRay supports a logging mode which allows the application to only capture a 228fixed amount of memory's worth of events. Flight Data Recorder (FDR) mode works 229very much like a plane's "black box" which keeps recording data to memory in a 230fixed-size circular queue of buffers, and have the data available 231programmatically until the buffers are finalized and flushed. To use FDR mode 232on your application, you may set the ``xray_mode`` variable to ``xray-fdr`` in 233the ``XRAY_OPTIONS`` environment variable. Additional options to the FDR mode 234implementation can be provided in the ``XRAY_FDR_OPTIONS`` environment 235variable. Programmatic configuration can be done by calling 236``__xray_log_init_mode("xray-fdr", <configuration string>)`` once it has been 237selected/installed. 238 239When the buffers are flushed to disk, the result is a binary trace format 240described by `XRay FDR format <XRayFDRFormat.html>`_ 241 242When FDR mode is on, it will keep writing and recycling memory buffers until 243the logging implementation is finalized -- at which point it can be flushed and 244re-initialised later. To do this programmatically, we follow the workflow 245provided below: 246 247.. code-block:: c++ 248 249 // Patch the sleds, if we haven't yet. 250 auto patch_status = __xray_patch(); 251 252 // Maybe handle the patch_status errors. 253 254 // When we want to flush the log, we need to finalize it first, to give 255 // threads a chance to return buffers to the queue. 256 auto finalize_status = __xray_log_finalize(); 257 if (finalize_status != XRAY_LOG_FINALIZED) { 258 // maybe retry, or bail out. 259 } 260 261 // At this point, we are sure that the log is finalized, so we may try 262 // flushing the log. 263 auto flush_status = __xray_log_flushLog(); 264 if (flush_status != XRAY_LOG_FLUSHED) { 265 // maybe retry, or bail out. 266 } 267 268The default settings for the FDR mode implementation will create logs named 269similarly to the basic log implementation, but will have a different log 270format. All the trace analysis tools (and the trace reading library) will 271support all versions of the FDR mode format as we add more functionality and 272record types in the future. 273 274 **NOTE:** We do not promise perpetual support for when we update the log 275 versions we support going forward. Deprecation of the formats will be 276 announced and discussed on the developers mailing list. 277 278Trace Analysis Tools 279-------------------- 280 281We currently have the beginnings of a trace analysis tool in LLVM, which can be 282found in the ``tools/llvm-xray`` directory. The ``llvm-xray`` tool currently 283supports the following subcommands: 284 285- ``extract``: Extract the instrumentation map from a binary, and return it as 286 YAML. 287- ``account``: Performs basic function call accounting statistics with various 288 options for sorting, and output formats (supports CSV, YAML, and 289 console-friendly TEXT). 290- ``convert``: Converts an XRay log file from one format to another. We can 291 convert from binary XRay traces (both basic and FDR mode) to YAML, 292 `flame-graph <https://github.com/brendangregg/FlameGraph>`_ friendly text 293 formats, as well as `Chrome Trace Viewer (catapult) 294 <https://github.com/catapult-project/catapult>` formats. 295- ``graph``: Generates a DOT graph of the function call relationships between 296 functions found in an XRay trace. 297- ``stack``: Reconstructs function call stacks from a timeline of function 298 calls in an XRay trace. 299 300These subcommands use various library components found as part of the XRay 301libraries, distributed with the LLVM distribution. These are: 302 303- ``llvm/XRay/Trace.h`` : A trace reading library for conveniently loading 304 an XRay trace of supported forms, into a convenient in-memory representation. 305 All the analysis tools that deal with traces use this implementation. 306- ``llvm/XRay/Graph.h`` : A semi-generic graph type used by the graph 307 subcommand to conveniently represent a function call graph with statistics 308 associated with edges and vertices. 309- ``llvm/XRay/InstrumentationMap.h``: A convenient tool for analyzing the 310 instrumentation map in XRay-instrumented object files and binaries. The 311 ``extract`` and ``stack`` subcommands uses this particular library. 312 313 314Minimizing Binary Size 315---------------------- 316 317XRay supports several different instrumentation points including ``function-entry``, 318``function-exit``, ``custom``, and ``typed`` points. These can be enabled individually 319using the ``-fxray-instrumentaton-bundle=`` flag. For example if you only wanted to 320instrument function entry and custom points you could specify: 321 322:: 323 324 clang -fxray-instrument -fxray-instrumentation-bundle=function-entry,custom ... 325 326This will omit the other sled types entirely, reducing the binary size. You can also 327instrument just a sampled subset of functions using instrumentation groups. 328For example, to instrument only a quarter of available functions invoke: 329 330:: 331 332 clang -fxray-instrument -fxray-function-groups=4 333 334A subset will be chosen arbitrarily based on a hash of the function name. To sample a 335different subset you can specify ``-fxray-selected-function-group=`` with a group number 336in the range of 0 to ``xray-function-groups`` - 1. Together these options could be used 337to produce multiple binaries with different instrumented subsets. If all you need is 338runtime control over which functions are being traced at any given time it is better 339to selectively patch and unpatch the individual functions you need using the XRay 340Runtime Library's ``__xray_patch_function()`` method. 341 342Future Work 343=========== 344 345There are a number of ongoing efforts for expanding the toolset building around 346the XRay instrumentation system. 347 348Trace Analysis Tools 349-------------------- 350 351- Work is in progress to integrate with or develop tools to visualize findings 352 from an XRay trace. Particularly, the ``stack`` tool is being expanded to 353 output formats that allow graphing and exploring the duration of time in each 354 call stack. 355- With a large instrumented binary, the size of generated XRay traces can 356 quickly become unwieldy. We are working on integrating pruning techniques and 357 heuristics for the analysis tools to sift through the traces and surface only 358 relevant information. 359 360More Platforms 361-------------- 362 363We're looking forward to contributions to port XRay to more architectures and 364operating systems. 365 366.. References... 367 368.. _`XRay whitepaper`: http://research.google.com/pubs/pub45287.html 369 370