1.. _module-pw_trace_tokenized:
2
3==================
4pw_trace_tokenized
5==================
6Pigweed's tracing module provides facilities for applications to trace
7information about the execution of their application. The module is split into
8two components:
9
101. The facade, provided elsewhere, which is only a macro interface layer
112. The backend (this module), is one implemention of the low level tracing.
12
13------
14Status
15------
16This module is currently in development, and is therefore still undergoing
17significant changes.
18
19Future work will add:
20
211. Add a more complete API for how to retrieve data from ring_buffer.
222. Add a Python library to decode the trace data.
233. Add examples with sample output (especially for filtering and triggering).
244. Add tools to retrieve trace data.
255. Add more sinks, such as RTT.
266. Add support to more platforms.
277. Improve the locking behaviour and provide default trace locking
28   implementions.
29
30--------
31Overview
32--------
33The tokenized trace backend aims to be a reasonable tradeoff of trace features
34and event size for most applications. It works by encoding all compile time data
35for a trace event into a tokenized number. This provides a good amount of
36compression, while maintaining the full trace feature set.
37
38In addition the tokenized trace backend adds flexibility through callbacks,
39which allows the application to do things such as filtering trace_events and
40triggering tracing to turn on and off. This flexibility can help maximize the
41effectiveness of a limited trace buffer as well as be a valuable tool while
42debugging.
43
44
45Compatibility
46-------------
47Most of this module is compatible with C and C++, the only exception to this is
48the ``RegisterCallbackWhenCreated`` helper class.
49
50Dependencies
51------------
52``pw_assert``
53``pw_log``
54``pw_preprocessor``
55``pw_status``
56``pw_tokenizer``
57``pw_trace:facade``
58``pw_varint``
59
60---------
61Macro API
62---------
63All code should use the trace API facade directly, this backend fully
64implements all features of the tracing facade.
65
66
67Event Callbacks & Data Sinks
68----------------------------
69The tokenized trace module adds both event callbacks and data sinks which
70provide hooks into tracing.
71
72The *event callbacks* are called when trace events occur, with the trace event
73data. Using the return flags, these callbacks can be used to adjust the trace
74behaviour at runtime in response to specific events. If requested (using
75``called_on_every_event``) the callback will be called on every trace event
76regardless if tracing is currently enabled or not. Using this, the application
77can trigger tracing on or off when specific traces or patterns of traces are
78observed, or can selectively filter traces to preserve the trace buffer.
79
80The event callback is a single function which is provided the details of the
81trace as arguments, and returns ``pw_trace_TraceEventReturnFlags``, which can be
82used to change how the trace is handled.
83
84.. cpp:function:: pw_trace_TraceEventReturnFlags pw_trace_EventCallback( \
85    void* user_data, \
86    uint32_t trace_ref, \
87    pw_trace_EventType event_type, \
88    const char* module, \
89    uint32_t trace_id, \
90    uint8_t flags)
91.. cpp:function:: pw_Status pw_trace_RegisterEventCallback( \
92    pw_trace_EventCallback callback, \
93    bool called_on_every_event, \
94    void* user_data, \
95    pw_trace_EventCallbackHandle* handle)
96.. cpp:function:: pw_Status pw_trace_UnregisterEventCallback( \
97    pw_trace_EventCallbackHandle handle)
98
99
100The *data sinks* are called only for trace events which get processed (tracing
101is enabled, and the sample not skipped). The sink callback is called with the
102encoded bytes of the trace event, which can be used by the application to
103connect different data sinks. The callback is broken into three callbacks
104``pw_trace_SinkStartBlock``, ``pw_trace_SinkAddBytes``, and
105``pw_trace_SinkEndBlock``. ``Start`` is called with the size of the block,
106before any bytes are emitted and can be used if needed to allocate space.
107``AddBytes`` is then called multiple times with chunks of bytes. Finally ``End``
108is called to allow any cleanup to be done by the sink if neccessary. Not all
109callbacks are required, it is acceptible to provide nullptr for any callbacks
110which you don't require.
111
112.. cpp:function:: void pw_trace_SinkStartBlock(void* user_data, size_t size)
113.. cpp:function:: void pw_trace_SinkAddBytes( \
114    void* user_data, \
115    const void* bytes, \
116    size_t size)
117.. cpp:function:: void pw_trace_SinkEndBlock(void* user_data)
118.. cpp:function:: pw_Status pw_trace_RegisterSink( \
119    pw_trace_SinkStartBlock start, \
120    pw_trace_SinkAddBytes add_bytes, \
121    pw_trace_SinkEndBlock end_block, \
122    void* user_data, \
123    pw_trace_SinkHandle* handle)
124.. cpp:function:: pw_Status pw_trace_UnregisterSink(pw_trace_SinkHandle handle)
125
126Trace Reference
127---------------
128Some use-cases might involve referencing a specific trace event, for example
129to use it as a trigger or filtering. Since the trace events are tokenized, a
130macro is provided to generate the token to use as a reference. All the fields
131must match exactly to generate the correct trace reference. If the trace does
132not have a group, use ``PW_TRACE_GROUP_LABEL_DEFAULT``.
133
134.. cpp:function:: PW_TRACE_REF(event_type, module, label, flags, group)
135.. cpp:function:: PW_TRACE_REF_DATA( \
136   event_type, module, label, flags, group, type)
137
138
139-----------
140Time source
141-----------
142Tracing rquires the platform to provide the time source for tracing, this can
143be done in one of a few ways.
144
1451. Create a file with the default time functions, and provide as build variable
146   ``pw_trace_tokenized_time``, which will get pulled in as a dependency.
1472. Provide time functions elsewhere in project, and ensure they are included.
1483. Redefine the trace time macros to something else, other then the default
149   trace time functions.
150
151.. cpp:function:: PW_TRACE_TIME_TYPE pw_trace_GetTraceTime()
152.. cpp:function:: PW_TRACE_GET_TIME()
153.. cpp:function:: size_t pw_trace_GetTraceTimeTicksPerSecond()
154.. cpp:function:: PW_TRACE_GET_TIME_TICKS_PER_SECOND()
155
156
157------
158Buffer
159------
160The optional trace buffer adds a ring buffer which contains the encoded trace
161data. This is still a work in progress, in particular better methods for
162retireving the data still needs to be added. Currently there is an accessor for
163the underlying ring buffer object, but this is a short term solution.
164
165.. cpp:function:: void ClearBuffer()
166.. cpp:function:: pw::ring_buffer::PrefixedEntryRingBuffer* GetBuffer()
167
168The buffer has two configurable options:
169
1701. PW_TRACE_BUFFER_SIZE_BYTES: The total size of the ring buffer in bytes.
1712. PW_TRACE_BUFFER_MAX_BLOCK_SIZE_BYTES: The maximum single trace object size.
172   Including the token, time, and any attached data. Any trace object larger
173   then this will be dropped.
174
175Added dependencies
176------------------
177``pw_ring_buffer``
178``pw_varint``
179
180
181-------
182Logging
183-------
184The optional trace buffer logging adds support to dump trace buffers to the log.
185Buffers are converted to base64-encoding then split across log lines. Trace logs
186are surrounded by 'begin' and 'end' tags.
187
188Ex. Invoking PW_TRACE_INSTANT with 'test1' and 'test2', then calling this
189function would produce this in the output logs:
190
191.. code:: sh
192
193  [TRACE] begin
194  [TRACE] data: BWdDMRoABWj52YMB
195  [TRACE] end
196
197Added dependencies
198------------------
199``pw_base64``
200``pw_log``
201``pw_ring_buffer``
202``pw_string``
203``pw_tokenizer``
204``pw_varint``
205
206--------
207Examples
208--------
209The examples all use `pw_trace` sample app to provide the trace data. Details
210for how to build, run, and decode the traces are included at the top of each
211example. This is early work, and is provided as an example of how different
212tracing concepts can look.
213
214Basic
215-----
216The basic example turns on tracing and dumps all trace output to a file provided
217on the command line.
218
219Trigger
220-------
221The trigger example demonstrates how a trace event can be used as a trigger to
222start and stop capturing a trace. The examples makes use of `PW_TRACE_REF` and
223`PW_TRACE_REF_DATA` to specify a start and stop event for the capture. This can
224be useful if the trace buffer is small and you wish to capture a specific
225series of events.
226
227Filter
228------
229The filter example demonstrates how a callback can be used to filter which trace
230events get processed and saved. In this example all events from the processing
231task which don't have traceId equal to 3 are removed. Both the other task traces
232are not removed. This can be a useful feature while debugging as it limits the
233amount of events which get stored to the buffer, and only saves the events of
234interest.
235