SkQP
====

SkQP (Skia Quality Program) is a component of the Android CTS (Compatablity
Test Suite) that tests an Android device's GPU and OpenGLES & Vulkan drivers
using Skia's existing unit & rendering tests.

See https://skia.org/dev/testing/skqp for pre-build APKs.

How to build and run the SkQP tests
-----------------------------------

1.  Get the dependencies:

    -   You will need Java JDK 8, `git`, and `python`.

    -   Install Chromium's [depot\_tools](http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html).  Add it to your `PATH`.

            git clone 'https://chromium.googlesource.com/chromium/tools/depot_tools.git'
            export PATH="${PWD}/depot_tools:${PATH}"

    -   Install the [Android NDK](https://developer.android.com/ndk/downloads/).

            ( cd ~; unzip ~/Downloads/android-ndk-*.zip )
            ANDROID_NDK=$(ls -d ~/android-ndk-*)   # Or wherever you installed the Android NDK.

    -   Install the [Android SDK](https://developer.android.com/studio/#command-tools).
        Set the `ANDROID_HOME` environment variable.

            mkdir ~/android-sdk
            ( cd ~/android-sdk; unzip ~/Downloads/sdk-tools-*.zip )
            yes | ~/android-sdk/tools/bin/sdkmanager --licenses
            export ANDROID_HOME=~/android-sdk  # Or wherever you installed the Android SDK.

        Put `adb` in your `PATH`.

            export PATH="${PATH}:${ANDROID_HOME}/platform-tools"

2.  Get the right version of Skia:

        git clone https://skia.googlesource.com/skia.git
        cd skia
        git checkout origin/skqp/dev  # or whatever release tag you need

3.  Build the APK:

        tools/git-sync-deps
        tools/skqp/make_universal_apk

4.  Build, install, and run.

        adb install -r out/skqp/skqp-universal-debug.apk
        adb logcat -c
        adb shell am instrument -w org.skia.skqp

5.  Monitor the output with:

        adb logcat TestRunner org.skia.skqp skia "*:S"

    Note the test's output path on the device.  It will look something like this:

        01-23 15:22:12.688 27158 27173 I org.skia.skqp:
        output written to "/storage/emulated/0/Android/data/org.skia.skqp/files/output"

6.  Retrieve and view the report with:

        OUTPUT_LOCATION="/storage/emulated/0/Android/data/org.skia.skqp/files/output"
        adb pull $OUTPUT_LOCATION /tmp/
        bin/sysopen /tmp/output/skqp_report/report.html

Running a single test
---------------------

To run a single test, for example `gles_aarectmodes`:

    adb shell am instrument -e class 'org.skia.skqp.SkQPRunner#gles_aarectmodes' -w org.skia.skqp

Unit tests can be run with the `unitTest_` prefix:

    adb shell am instrument -e class 'org.skia.skqp.SkQPRunner#unitTest_GrSurface -w org.skia.skqp

Run as a non-APK executable
---------------------------

1.  Follow steps 1-3 as above.

2.  Build the SkQP program, load files on the device, and run skqp:

        ninja -C out/skqp/arm skqp
        python tools/skqp/run_skqp_exe out/skqp/arm

SkQP Render Test Algorithm
==========================

The following is a description of the render test validation algorithm that
will be used by the version of SkQP that will be released for Android Q-release.

There is a global macro constant: `SK_SKQP_GLOBAL_ERROR_TOLERANCE`, which
reflects the `gn` variable `skia_skqp_global_error_tolerance`.  This is usually
set to 8.

First, look for a file named `skqp/rendertests.txt` in the
`platform_tools/android/apps/skqp/src/main/assets` directory.  The format of
this file is:  each line contains one render test name, followed by a comma,
followed by an integer.  The integer is the `passing_threshold` for that test.

For each test, we have a `max_image` and a `min_image`.  These are PNG-encoded
images stored in SkQP's APK's asset directory (in the paths `gmkb/${TEST}/min.png`
and `gmkb/${TEST}/max.png`).

The test input is a rendered image.  This will be produced by running one of
the render tests against the either the `vk` (Vulkan) or `gles` (OpenGL ES)
Skia backend.

Here is psuedocode for the error calculation:

    function calculate_pixel_error(pixel_value, pixel_max, pixel_min):
        pixel_error = 0

        for color_channel in { red, green, blue, alpha }:
            value = get_color(pixel_value, color_channel)
            v_max = get_color(pixel_max,   color_channel)
            v_min = get_color(pixel_min,   color_channel)

            if value > v_max:
                channel_error = value - v_max
            elif value < v_min:
                channel_error = v_min - value
            else:
                channel_error = 0
            pixel_error = max(pixel_error, channel_error)

        return max(0, pixel_error - SK_SKQP_GLOBAL_ERROR_TOLERANCE);

    function get_error(rendered_image, max_image, min_image):
        assert(dimensions(rendered_image) == dimensions(max_image))
        assert(dimensions(rendered_image) == dimensions(min_image))

        max_error = 0
        bad_pixels = 0
        total_error = 0

        error_image = allocate_bitmap(dimensions(rendered_image))

        for xy in list_all_pixel_coordinates(rendered_image):
            pixel_error = calculate_pixel_error(rendered_image(xy),
                                                max_image(xy),
                                                min_image(xy))
            if pixel_error > 0:
                for neighboring_xy in find_neighbors(xy):
                    if not inside(neighboring_xy, dimensions(rendered_image)):
                        continue
                    pixel_error = min(pixel_error,
                                      calculate_pixel_error(rendered_image(xy),
                                                            max_image(neighboring_xy),
                                                            min_image(neighboring_xy)))

            if pixel_error > 0:
                max_error = max(max_error, pixel_error)
                bad_pixels += 1
                total_error += pixel_error

                error_image(xy) = linear_interpolation(black, red, pixel_error)
            else:
                error_image(xy) = white

        return ((total_error, max_error, bad_pixels), error_image)

For each render test, there is a threshold value for `total_error`, :
`passing_threshold`.

If `passing_threshold >= 0 && total_error > passing_threshold`, then the test
is a failure and is included in the report.  if `passing_threshold == -1`, then
the test always passes, but we do execute the test to verify that the driver
does not crash.

We generate a report with the following information for each test:

    backend_name,render_test_name,max_error,bad_pixels,total_error

in CSV format in the file `out.csv`.  A HTML report of just the failing tests
is written to the file `report.html`.  This version includes four images for
each test:  `rendered_image`, `max_image`, `min_image`, and `error_image`, as
well as the three metrics: `max_error`, `bad_pixels`, and `total_error`.

How SkQP Generates Render Test Models
=====================================

We will, at regular intervals, generate new models from the [master branch of
Skia][1].  Here is how that process works:

0.  Choose a commit to make the branch from

        COMMIT=origin/master

    Or use the script to find the best one:

        cd SKIA_SOURCE_DIRECTORY
        git fetch origin
        COMMIT=$(python tools/skqp/find_commit_with_best_gold_results.py \
                 origin/master ^origin/skqp/dev)

1.  Get the positively triaged results from Gold and generate models:

        cd SKIA_SOURCE_DIRECTORY
        git fetch origin
        git checkout "$COMMIT"
        python tools/skqp/cut_release.py HEAD~10 HEAD

    This will create the following files:

        platform_tools/android/apps/skqp/src/main/assets/files.checksum
        platform_tools/android/apps/skqp/src/main/assets/skqp/rendertests.txt
        platform_tools/android/apps/skqp/src/main/assets/skqp/unittests.txt

    These three files can be commited to Skia to create a new commit.  Make
    `origin/skqp/dev` a parent of this commit (without merging it in), and
    push this new commit to `origin/skqp/dev`, using this script:

        sh tools/skqp/branch_skqp_dev.sh

    Review and submit the change:

        git push origin HEAD:refs/for/skqp/dev
        bin/sysopen https://review.skia.org/$(bin/gerrit-number HEAD)

    (Optional) Make a SkQP APK.

        tools/skqp/docker_build_universal_apk.sh

    (Optional) Test the SkQP APK:

        tools/skqp/test_apk.sh (LOCATION)/skqp-universal-debug.apk

    (Once changes land) Upload the SkQP APK.

        tools/skqp/upload_apk HEAD (LOCATION)/skqp-universal-debug.apk


`tools/skqp/cut_release.py`
---------------------------

This tool will call `make_skqp_model` to generate the `m{ax,in}.png` files for
each render test.

Then it calls `jitter_gms` to see which render tests pass the jitter test.
`jitter_gms` respects the `bad_gms.txt` file by ignoring the render tests
enumerated in that file.  Tests which pass the jitter test are enumerated in
the file `good.txt`, those that fail in the `bad.txt` file.

Next, the `skqp/rendertests.txt` file is created.  This file lists the render
tests that will be executed by SkQP.  These are the union of the tests
enumerated in the `good.txt` and `bad.txt` files.  If the render test is found
in the `good.txt` file and the model exists, its per-test threshold is set
to 0 (a later CL can manually change this, if needed).  Otherwise, the
threshold is set to -1; this indicated that the rendertest will be executed (to
verify that the driver will not crash), but the output will not be compared
against the model.  Unnecessary models will be removed.

Next, all of the files that represent the models are uploaded to cloud storage.
A single checksum hash is kept in the  `files.checksum` file.  This is enough
to re-download those files later, but we don't have to fill the git repository
with a lot of binary data.

Finally, a list of the current gpu unit tests is created and stored in
`skqp/unittests.txt`.

[1]: https://skia.googlesource.com/skia/+log/master "Skia Master Branch"
[2]: https://gold.skia.org/search                   "Skia Gold Search"