1page.title=Automating the tests 2@jd:body 3 4<!-- 5 Copyright 2015 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18--> 19 20<div id="qv-wrapper"> 21 <div id="qv"> 22 <h2>In this document</h2> 23 <ol id="auto-toc"> 24 </ol> 25 </div> 26</div> 27 28<h2 id=intro>Introduction</h2> 29 30<p>Deqp test modules can be integrated to automated test systems in multiple ways. 31The best approach depends on the existing test infrastructure and target 32environment.</p> 33 34<p>The primary output from a test run is always the test log file, i.e. the file 35with a <code>.qpa</code> postfix. Full test results can be parsed from the test log. Console output is 36debug information only and may not be available on all platforms.</p> 37 38<p>Test binaries can be invoked directly from a test automation system. The test 39binary can be launched for a specific case, for a test set, or for all 40available tests. If a fatal error occurs during execution (such as certain API 41errors or a crash), the test execution will abort. For regression testing, the 42best approach is to invoke the test binaries for individual cases or small test 43sets separately, in order to have partial results available even in the event 44of hard failure.</p> 45 46<p>The deqp comes with command line test execution tools that can be used in 47combination with the execution service to achieve a more robust integration. 48The executor detects test process termination and will resume test execution on 49the next available case. A single log file is produced from the full test 50session. This setup is ideal for lightweight test systems that don’t provide 51crash recovery facilities.</p> 52 53<h2 id=command_line_test_execution_tools>Command line test execution tools</h2> 54 55<p>The current command line tool set includes a remote test execution tool, a test 56log comparison generator for regression analysis, a test-log-to-CSV converter, 57a test-log-to-XML converter, and a testlog-to-JUnit converter.</p> 58 59<p>The source code for these tools is in the <code>executor</code> directory, and the binaries are built into the <code><builddir>/executor</code> directory.</p> 60 61<h3 id=command_line_test_executor>Command line Test Executor</h3> 62 63<p>The command line Test Executor is a portable C++ tool for launching a test run 64on a device and collecting the resulting logs from it over TCP/IP. The Executor 65communicates with the execution service (execserver) on the target device. 66Together they provide functionality such as recovery from test process crashes. 67The following examples demonstrate how to use the command line Test Executor 68(use <code>--help</code> for more details):</p> 69 70<h4 id=example_1_run_gles2_functional_tests>Example 1: Run GLES2 functional tests on an Android device:</h4> 71 72<pre> 73executor --connect=127.0.0.1 --port=50016 --binaryname= 74com.drawelements.deqp/android.app.NativeActivity 75--caselistdir=caselists 76--testset=dEQP-GLES2.* --out=BatchResult.qpa 77--cmdline="--deqp-crashhandler=enable --deqp-watchdog=enable 78--deqp-gl-config-name=rgba8888d24s8" 79</pre> 80 81<h4 id=example_2_continue_a_partial_opengl>Example 2: Continue a partial OpenGL ES 2 test run locally:</h4> 82 83<pre> 84executor --start-server=execserver/execserver --port=50016 85--binaryname=deqp-gles2 --workdir=modules/opengl 86--caselistdir=caselists 87--testset=dEQP-GLES2.* --exclude=dEQP-GLES2.performance.* --in=BatchResult.qpa 88--out=BatchResult.qpa 89</pre> 90 91<h3 id=test_log_csv_export_and_compare>Test log CSV export and compare</h3> 92 93<p>The deqp has a tool for converting test logs (.<code>qpa </code>files) into CSV files. The CSV output contains a list of test cases and their 94results. The tool can also compare two or more batch results and list only the 95test cases that have different status codes in the input batch results. The 96comparison will also print the number of matching cases.</p> 97 98<p>The output in CSV format is very practical for further processing with standard 99command line utilities or with a spreadsheet editor. An additional, human-readable, 100plain-text format can be selected using the following command line argument: <code>--format=text</code></p> 101 102<h4 id=example_1_export_test_log_in_csv_format>Example 1: Export test log in CSV format</h4> 103 104<pre> 105testlog-to-csv --value=code BatchResult.qpa > Result_statuscodes.csv 106testlog-to-csv --value=details BatchResult.qpa > Result_statusdetails.csv 107</pre> 108 109<h4 id=example_2_list_differences>Example 2: List differences of test results between two test logs</h4> 110 111<pre> 112testlog-to-csv --mode=diff --format=text Device_v1.qpa Device_v2.qpa 113</pre> 114 115<p class="note"><strong>Note:</strong> The argument <code>--value=code</code> outputs the test result code, such as "Pass" or "Fail". The argument <code>--value=details</code> selects the further explanation of the result or numerical value produced by a performance, capability, or accuracy test.</p> 116 117<h3 id=test_log_xml_export>Test log XML export</h3> 118 119<p>Test log files can be converted to valid XML documents using the <code>testlog-to-xml</code> utility. Two output modes are supported: </p> 120 121<ul> 122 <li> Separate documents mode, where each test case and the <code>caselist.xml</code> summary document are written to a destination directory 123 <li> Single file mode, where all results in the <code>.qpa</code> file are written to single XML document. 124</ul> 125 126<p>Exported test log files can be viewed in a browser using an XML style sheet. 127Sample style sheet documents (<code>testlog.xsl</code> and <code>testlog.css</code>) are provided in the <code>doc/testlog-stylesheet</code> directory. To render the log files in a browser, copy the two style sheet 128files into the same directory where the exported XML documents are located.</p> 129 130<p>If you are using Google Chrome, the files must be accessed over HTTP as Chrome 131limits local file access for security reasons. The standard Python installation 132includes a basic HTTP server that can be launched to serve the current 133directory with the <code>python –m SimpleHTTPServer 8000</code> command. After launching the server, just point the Chrome browser to <code>http://localhost:8000</code> to view the test log.</p> 134 135<h3 id=conversion_to_a_junit_test_log>Conversion to a JUnit test log</h3> 136 137<p>Many test automation systems can generate test run result reports from JUnit 138output. The deqp test log files can be converted to the JUnit output format 139using the testlog-to-junit tool. </p> 140 141<p>The tool currently supports translating the test case verdict only. As JUnit 142supports only "pass" and "fail" results, a passing result of the deqp is mapped 143to "JUnit pass" and other results are considered failures. The original deqp 144result code is available in the JUnit output. Other data, such as log messages 145and result images, are not preserved in the conversion.</p> 146