1# Flicker Test Library
2
3## Motivation
4Detect *flicker* — any discontinuous, or unpredictable behavior seen during UI transitions that is not due to performance.
5This is often the result of a logic error in the code and difficult to identify because the issue is transient and at times difficult to reproduce.
6This library helps create integration tests between `SurfaceFlinger`, `WindowManager` and `SystemUI` to identify flicker.
7
8Examples of flickers are:
9* Elements drawn in the wrong location (e.g., not rotating correctly)
10* Empty areas on the screen (e.g., not all areas of the screen covered during rotation)
11* System elements not appearing (e.g., nav and status bar or split screen divider)
12* Elements (dis)appearing at the wrong time (e.g., an element becomes invisible before being completely occluded)
13
14## Usage and Capabilities
15
16The library builds and runs UI transitions, captures Winscope [traces](https://source.android.com/devices/graphics/tracing-win-transitions) and exposes common assertions that can be tested against each trace.
17
18## Building a transition
19
20Start by defining common or error prone transitions using `TransitionRunner`.
21```java
22// Example: Build a transition that cold launches an app from launcher
23TransitionRunner transition = new TransitionBuilder()
24                // Specify a tag to identify the transition (optional)
25                .withTag("OpenAppCold_" + testApp.getLauncherName())
26
27                // Specify preconditions to setup the device
28                // Wake up device and go to home screen
29                .runBeforeAll(AutomationUtils::wakeUpAndGoToHomeScreen)
30
31                // Setup transition under test
32                // Press the home button and close the app to test a cold start
33                .runBefore(device::pressHome)
34                .runBefore(testApp::exit)
35
36                // Run the transition under test
37                // Open the app and wait for UI to be idle
38                // This is the part of the transition that will be tested.
39                .run(testApp::open)
40                .run(device::waitForIdle)
41
42                // Perform any tear downs
43                // Close the app
44                .runAfterAll(testApp::exit)
45
46                // Number of times to repeat the transition to catch any flaky issues
47                .repeat(5);
48```
49
50Run the transition to get a list of `TransitionResult` for each time the transition is repeated.
51```java
52    List<TransitionResult> results = transition.run();
53```
54`TransitionResult` contains paths to test artifacts such as Winscope traces and screen recordings.
55
56### Checking Assertions
57Each `TransitionResult` can be tested using an extension of the Google Truth library, `LayersTraceSubject` and `WmTraceSubject`.
58They try to balance test principles set out by Google Truth (not supporting nested assertions, keeping assertions simple) with providing support for common assertion use cases.
59
60Each trace can be represented as a ordered collection of trace entries, with an associated timestamp.
61Each trace entry has common assertion checks.
62The trace subjects expose methods to filter the range of entries and test for changing assertions.
63
64```java
65    TransitionResult result = results.get(0);
66    Rect displayBounds = getDisplayBounds();
67
68    // check all trace entries
69    assertThat(result).coversRegion(displayBounds).forAllEntries();
70
71    // check a range of entries
72    assertThat(result).coversRegion(displayBounds).forRange(startTime, endTime);
73
74    // check first entry
75    assertThat(result).coversRegion(displayBounds).inTheBeginning();
76
77    // check last entry
78    assertThat(result).coversRegion(displayBounds).atTheEnd();
79
80    // check a change in assertions, e.g. wallpaper window is visible,
81    // then wallpaper window becomes and stays invisible
82    assertThat(result)
83                .showsBelowAppWindow("wallpaper")
84                .then()
85                .hidesBelowAppWindow("wallpaper")
86                .forAllEntries();
87```
88
89All assertions return `Result` which contains a `success` flag, `assertionName` string identifier, and `reason` string to provide actionable details to the user.
90The `reason` string is build along the way with all the details as to why the assertions failed and any hints which might help the user determine the root cause.
91Failed assertion message will also contain a path to the trace that was tested.
92Example of a failed test:
93
94```
95    java.lang.AssertionError: Not true that <com.android.server.wm.traces.common.LayersTrace@65da4cc>
96    Layers Trace can be found in: /layers_trace_emptyregion.pb
97    Timestamp: 2308008331271
98    Assertion: coversRegion
99    Reason:   Region to test: Rect(0, 0 - 1440, 2880)
100    first empty point: 0, 99
101    visible regions:
102    StatusBar#0Rect(0, 0 - 1440, 98)
103    NavigationBar#0Rect(0, 2712 - 1440, 2880)
104    ScreenDecorOverlay#0Rect(0, 0 - 1440, 91)
105    ...
106        at com.google.common.truth.FailureStrategy.fail(FailureStrategy.java:24)
107        ...
108```
109
110---
111
112## Running Tests
113
114The tests can be run as any other Android JUnit tests. `frameworks/base/tests/FlickerTests` uses the library to test common UI transitions. Run `atest FlickerTests` to execute these tests.
115
116---
117
118## Other Topics
119### Monitors
120Monitors capture test artifacts for each transition run. They are started before each iteration of the test transition (after the `runBefore` calls) and stopped after the transition is completed. Each iteration will produce a new test artifact. The following monitors are available:
121
122#### LayersTraceMonitor
123Captures Layers trace. This monitor is started by default. Build a transition with `skipLayersTrace()` to disable this monitor.
124#### WindowManagerTraceMonitor
125Captures Window Manager trace. This monitor is started by default. Build a transition with `skipWindowManagerTrace()` to disable this monitor.
126#### WindowAnimationFrameStatsMonitor
127Captures WindowAnimationFrameStats for the transition. This monitor is started by default and is used to eliminate *janky* runs. If an iteration has skipped frames, as determined by WindowAnimationFrameStats, the results for the iteration is skipped. If the list of results is empty after all iterations are completed, then the test should fail. Build a transition with `includeJankyRuns()` to disable this monitor.
128#### ScreenRecorder
129Captures screen to a video file. This monitor is disabled by default. Build a transition with `recordEachRun()` to capture each transition or build with `recordAllRuns()` to capture every transition including setup and teardown.
130
131---
132
133### Extending Assertions
134
135To add a new assertion, add a function to one of the trace entry classes, `LayersTrace.Entry` or `WindowManagerTrace.Entry`.
136
137```java
138    // Example adds an assertion to the check if layer is hidden by parent.
139    Result isHiddenByParent(String layerName) {
140        // Result should contain a details if assertion fails for any reason
141        // such as if layer is not found or layer is not hidden by parent
142        // or layer has no parent.
143        // ...
144    }
145```
146Then add a function to the trace subject `LayersTraceSubject` or `WmTraceSubject` which will add the assertion for testing. When the assertion is evaluated, the trace will first be filtered then the assertion will be applied to the remaining entries.
147
148```java
149    public LayersTraceSubject isHiddenByParent(String layerName) {
150        mChecker.add(entry -> entry.isHiddenByParent(layerName),
151                "isHiddenByParent(" + layerName + ")");
152        return this;
153    }
154```
155
156To use the new assertion:
157```java
158    // Check if "Chrome" layer is hidden by parent in the first trace entry.
159    assertThat(result).isHiddenByParent("Chrome").inTheBeginning();
160```