1dEQP
2====
3
4These steps are specifically for testing SwiftShader's OpenGL ES 3.0 implementation using dEQP on Windows (steps for Linux below the Windows instructions).
5
6Prerequisites
7-------------
8
91. Install the latest [Python 3](https://www.python.org/downloads/)
102. Install [Visual Studio](https://visualstudio.microsoft.com/vs/community/)
113. Install [CMake](https://cmake.org/download/)
124. Install [Go](https://golang.org/doc/install)
135. Install [MinGW-W64](http://mingw-w64.org/doku.php/download)
14  * Select 'x86_64' as Architecture during setup
156. Install [Git](https://git-scm.com/download/win)
167. Install [Android Studio](https://developer.android.com/studio/index.html)
178. Run Android Studio and install Android SDK.
189. Set environment variables: Config Panel -> System and Security -> System -> Advanced system settigns -> Environment Variables
19  * Add `<path to python>` to your PATH environment variable
20  * Add `<path to MinGW-W64>\bin` to your PATH environment variable
21  * Add `<path to adb>` to your PATH environment variable
22
23    Note: abd is in the Android SDK, typically in `C:\Users\<username>\AppData\Local\Android\sdk\platform-tools`
24
2511. (Optional) Install [TortoiseGit](https://tortoisegit.org/)
26
27Getting the Code
28----------------
29
3012. Get dEQP (either in 'cmd' or by using TortoiseGit):
31
32    `git clone https://github.com/KhronosGroup/VK-GL-CTS`
33
34    You may wish to check out a stable vulkan-cts-* branch.
35
3613. Get dEQP's dependencies. In your dEQP root directory, open 'cmd' and run:
37
38    `python3 external\fetch_sources.py`
39
4014. Get Cherry (either in 'cmd' or by using TortoiseGit):
41
42    `git clone https://android.googlesource.com/platform/external/cherry`
43
4415. Set environment variable (see point 9):
45
46    Add new variable GOPATH='`<path to cherry>`'
47
48Building the code
49-----------------
50
5116. Build dEQP's Visual Studio files using the CMake GUI, or, in the dEQP root dir, run:
52    ```
53    mkdir build
54    cd build
55    cmake ..
56    ```
57    Note: don't call 'cmake .' directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.
58
5917. Build dEQP:
60
61    Open `<path to dEQP>\build\dEQP-Core-default.sln` in Visual Studio and Build Solution
62
63    Note: Choose a 'Release' build, unless you really mean to debug dEQP
64
6518. Generate test cases:
66    ```
67    mkdir <path to cherry>\data
68    cd <path to dEQP>
69    python3 scripts\build_caselists.py <path to cherry>\data
70    ```
71
72    Note: you need to run `python3 scripts\build_caselists.py <path to cherry>\data` every time you update dEQP.
73
74Preparing the server
75--------------------
76
7719. Edit `<path to cherry>\cherry\data.go`
78* Search for `../candy-build/deqp-wgl` and replace that by `<path to deqp>/build`
79* Just above, add an option to CommandLine: `--deqp-gl-context-type=egl`
80* Just below, modify the BinaryPath from 'Debug' to 'Release' if you did a Release build at step 17
81
82Testing OpenGL ES
83-----------------
84
8520. a) Assuming you already built SwiftShader in the `build` folder, copy these two files:
86
87    `libEGL.dll`\
88    `libGLESv2.dll`
89
90    From:
91
92    `<path to SwiftShader>\build\Release_x64` or\
93    `<path to SwiftShader>\build\Debug_x64`
94
95    To:
96
97    `<path to dEQP>\build\modules\gles3\Release` (Again, assuming you did a Release build at step 17)
98
99Testing Vulkan
100--------------
101
10220. b) Assuming you already built SwiftShader, copy and rename this file:
103
104    `<path to SwiftShader>\build\Release_x64\vk_swiftshader.dll` or\
105    `<path to SwiftShader>\build\Debug_x64\vk_swiftshader.dll`
106
107    To:
108
109    `<path to dEQP>\build\external\vulkancts\modules\vulkan\vulkan-1.dll`
110
111    This will cause dEQP to load SwiftShader's Vulkan implementatin directly, without going through a system-provided [loader](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#the-loader) library or any layers.
112
113    To use SwiftShader as an [Installable Client Driver](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers) (ICD) instead:
114    * Edit environment variables:
115      * Define VK_ICD_FILENAMES to `<path to SwiftShader>\src\Vulkan\vk_swiftshader_icd.json`
116    * If the location of `vk_swiftshader.dll` you're using is different than the one specified in `src\Vulkan\vk_swiftshader_icd.json`, modify it to point to the `vk_swiftshader.dll` file you want to use.
117
118Running the tests
119-----------------
120
12121. Start the test server. Go to `<path to cherry>` and run:
122
123    `go run server.go`
124
12522. Open your favorite browser and navigate to `localhost:8080`
126
127    Get Started -> Choose Device 'localhost' -> Select Tests 'dEQP-GLES3' -> Execute tests!
128
129Mustpass sets
130-------------
131
132dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The [android\cts\master\gles3-master.txt](https://android.googlesource.com/platform/external/deqp/+/master/android/cts/master/gles3-master.txt) text file which can be loaded in Cherry's 'Test sets' tab to only run the latest tests expected to pass by certified Android devices.
133
134Linux
135-----
136
137The Linux process is similar to Windows. However it doesn't use Release or Debug variants and it uses shared object files instead of DLLs.
138
1391. Install the latest [Python 3](https://www.python.org/downloads/)
1402. Install GCC and Make. In a terminal, run:
141
142    `sudo apt-get install gcc make`
143
1443. Install [CMake](https://cmake.org/download/)
1454. Install [Go](https://golang.org/doc/install)
1465. Install Git. In a terminal, run:
147
148    `sudo apt-get install git`
149
1506. Download the [Vulkan SDK](https://vulkan.lunarg.com/) and unpack it into a location you like.
151
152Getting the Code
153----------------
154
1557. Get Swiftshader. In a terminal, go to the location you want to keep Swiftshader, and run:
156
157    ```
158    git clone https://swiftshader.googlesource.com/SwiftShader && (cd SwiftShader && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit-review.googlesource.com/tools/hooks/commit-msg ; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
159    ```
160
161    This will also install the commit hooks you need for committing to SwiftShader.
162
1638. Get dEQP:
164
165    `git clone sso://googleplex-android/platform/external/deqp`
166
1679. Get dEQP's dependencies. In your dEQP root directory, run:
168
169    `python3 external/fetch_sources.py`
170
17110. Get Cherry, similar to step 8:
172
173    `git clone https://android.googlesource.com/platform/external/cherry`
174
17511. Set environment variable. Open ~/.bashrc in your preferred editor and add the following line:
176
177    GOPATH='`<path to cherry>`'
178
179Building the code
180-----------------
181
18212. Build Swiftshader. In the Swiftshader root dir, run:
183    ```
184    cd build
185    cmake ..
186    make --jobs=$(nproc)
187    ```
188
18913. Set your environment variables. In the terminal in which you'll be building dEQP, run the following commands:
190
191    ```
192    export LD_LIBRARY_PATH="<Vulkan SDK location>/x86_64/lib:$LD_LIBRARY_PATH"
193    export LD_LIBRARY_PATH="<Swiftshader location>/build:$LD_LIBRARY_PATH"
194    ```
195
196    It's important that you perform this step before you build dEQP in the next step. CMake will search for library files in LD_LIBRARY_PATH. If it cannot discover Swiftshader's libEGL and libGLESv2 shared object files, then CMake will default to using your system's libEGL.so and libGLESv2.so files.
197
19814. Build dEQP. In the dEQP root dir, run:
199    ```
200    mkdir build
201    cd build
202    cmake ..
203    make --jobs=$(nproc)
204    ```
205
206    Also: don't call 'cmake .' directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.
207
20815. Generate test cases:
209    ```
210    mkdir <path to cherry>/data
211    cd <path to dEQP>
212    python3 scripts/build_caselists.py <path to cherry>/data
213    ```
214
215    Note: you need to run `python3 scripts/build_caselists.py <path to cherry>/data` every time you update dEQP.
216
217Preparing the server
218--------------------
219
22016. Edit `<path to cherry>/cherry/data.go`
221* Search for ".exe" and remove all instances.
222* Search for `../candy-build/deqp-wgl/execserver/Release` and replace that by `<path to deqp>/build/execserver/`
223* Just above, add an option to CommandLine: `--deqp-gl-context-type=egl`
224* Just below, remove 'Debug/' from the BinaryPath.
225* Just one more line below, replace `../candy-build/deqp-wgl/` with `<path to deqp>/build`.
226
227Testing OpenGL ES
228-----------------
229
23017. a) Assuming you setup the LD_LIBRARY_PATH environment variable prior to running CMake in the dEQP build directory, you're all set.
231
232Testing Vulkan
233--------------
234
23517. b) Use SwiftShader as an [Installable Client Driver](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers) (ICD). Add the following line to your `~/.bashrc`:
236
237      `export VK_ICD_FILENAMES="<path to SwiftShader>/build/Linux/vk_swiftshader_icd.json"`
238
239    Then run `source ~/.bashrc` the terminal(s) you'll be running tests from.
240
241
242Running the tests
243-----------------
244
24518. Start the test server. Go to `<path to cherry>` and run:
246
247    `go run server.go`
248
24919. Open your favorite browser and navigate to `localhost:8080`
250
251    Get Started -> Choose Device 'localhost' -> Select Tests 'dEQP-GLES3' -> Execute tests!
252
25320. To make sure that you're running SwiftShader's drivers, select only the dEQP-GLES3->info->vendor and dEQP-VK->info->platform tests. In the next window, click on these tests in the left pane. If you see Google inc for the GLES3 test and your Linux machine in the VK test, then you've set your suite up properly.
254
25521. If you want to run Vulkan tests in the command line, go to the build directory in dEQP root. Then run the following command:
256
257    `external/vulkanacts/modules/vulkan/deqp-vk`
258
259    You can also run individual tests with:
260
261    `external/vulkanacts/modules/vulkan/deqp-vk --deqp-case=<test name>`
262
263    And you can find a list of the test names in `<Swiftshader root>/tests/regres/testlists/vk-master.txt` However, deqp-vk will cease upon the first failure. It's recommended that you use cherry for your testing needs unless you know what you're doing.
264
26522. To check that you're running SwiftShader in cherry, start the server
266
267Mustpass sets
268-------------
269
270dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The [android\cts\master\gles3-master.txt](https://android.googlesource.com/platform/external/deqp/+/master/android/cts/master/gles3-master.txt) text file which can be loaded in Cherry's 'Test sets' tab to only run the latest tests expected to pass by certified Android devices.
271