This tutorial guides you through the construction of a "hello world" Trade Federation test configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting from the TF development environment, it guides you through the process of creating a simple Trade Federation config and gradually adding more features to it.
The tutorial presents the TF test development process as a set of exercises, each consisting of several steps. The exercises demonstrate how to gradually build and refine your configuration, and provide all the sample code you need to complete the test configuration. The title of each exercise is annotated with a letter describing which roles are involved in that step: D for Developer, I for Integrator, and/or R for Test Runner.
When you are finished with the tutorial, you will have created a functioning TF configuration and will have learned many of the most important concepts in the TF framework.
See the Machine Setup page for how to setup the development environment. The rest of this tutorial assumes you have a shell open that has been initialized to the Trade Federation environment.
For simplicity, this tutorial will illustrate adding a configuration and its classes to the Trade Federation framework core library. This can be extended to developing modules outside the source tree by simply compiling the tradefed JAR, and compiling your modules against that JAR.
Lets create a hello world test that just dumps a message to stdout. A tradefed test will generally implement the IRemoteTest interface.
Here's an implementation for the HelloWorldTest:
package com.android.tradefed.example;
import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;
public class HelloWorldTest implements IRemoteTest {
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World!");
}
}
Save this sample code to
<tree>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java
and rebuild tradefed from your shell:
m -jNIf the build does not succeed, consult the Machine Setup page to ensure that you didn't miss any steps.
Trade Federation tests are made executable by creating a Configuration, which is an XML file that instructs tradefed on which test (or tests) to run, as well as which other modules to execute, and in what order.
Lets create a new Configuration for our HelloWorldTest:
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
</configuration>TF will parse the Configuration XML file (aka config), load the specified class using
reflection, instantiate it, cast it to a IRemoteTest, and call its run
method.
Note that we've specified the full class name of the HelloWorldTest. Save this data to a
helloworld.xml file anywhere on your local filesystem (eg /tmp/helloworld.xml).
From your shell, launch the tradefed console
$ tradefed.sh
Ensure that a device is connected to the host machine and is visible to tradefed
tf >list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
Configurations can be executed using the run <config> console command. Try this:
FIXME: redo this
tf> run /tmp/helloworld.xml
05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
You should see "Hello, TF World!" outputted on the terminal.
For convenience of deployment, you can also bundle configs into the tradefed jars themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on the classpath.
Lets illustrate this now by moving the helloworld.xml into the tradefed core library.
Move the helloworld.xml file into
<tree>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml.
Rebuild tradefed, and restart the tradefed console.
Ask tradefed to display the list of configurations from the classpath:
tf> list configs
[…]
example/helloworld: Runs the hello world test
You can now run the helloworld config via the following command
tf >run example/helloworld
05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
So far our hello world test isn't doing anything interesting. Tradefed's specialty is running tests using Android devices, so lets add an Android device to the test.
Tests can get a reference to an Android device by implementing the IDeviceTest interface.
Here's a sample implementation of what this looks like:
public class HelloWorldTest implements IRemoteTest, IDeviceTest {
private ITestDevice mDevice;
@Override
public void setDevice(ITestDevice device) {
mDevice = device;
}
@Override
public ITestDevice getDevice() {
return mDevice;
}
…
}
The Trade Federation framework will inject the ITestDevice reference into your
test via the IDeviceTest#setDevice method, before the IRemoteTest#run
method is called.
Let's modify the HelloWorldTest print message to display the serial number of the device.
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
}
Now rebuild tradefed, and check the list of devices:
$ tradefed.sh
tf >list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
Take note of the serial number listed as Available above. That is the device that should be allocated to HelloWorld.
tf >run example/helloworld
05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
You should see the new print message displaying the serial number of the device.
IRemoteTests report results by calling methods on the
ITestInvocationListener instance provided to their #run method. Note that the
TF framework itself is responsible for reporting the start and end of each Invocation, (via
the ITestInvocationListener#invocationStarted and
ITestInvocationListener#invocationEnded methods, respectively).
A test run is a logical collection of tests. To report test results,
IRemoteTests are responsible
for reporting the start of a test run, the start and end of each test, and the end of the test run.
Here's what the HelloWorldTest implementation might look like with a single failed test result.
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest");
listener.testRunStarted("helloworldrun", 1);
listener.testStarted(testId);
listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed");
listener.testEnded(testId, Collections.emptyMap());
listener.testRunEnded(0, Collections.emptyMap());
}Note that Trade Federation also includes several IRemoteTest implementations that
you can reuse instead of writing your own from scratch. These include, for instance,
InstrumentationTest, which can run an Android application's tests remotely on an Android
device, parse the results, and forward them to the ITestInvocationListener). See the
Test Types
documentation for more details.
By default, a TF config will use the
TextResultReporter as
the test listener implementation. TextResultReporter will dump the results of an
invocation to stdout. To illustrate, try running the hello-world config from the previous
section:
$ ./tradefed.sh
tf >run example/helloworld
05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests
Test FAILURE: com.example.TestClassName#sampleTest
stack: oh noes, test failed
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms
If you want to store the results of an invocation elsewhere, such as in a file, you need to
specify a custom ITestInvocationListener implementation by using the
result_reporter tag in your configuration.
Trade Federation includes the XmlResultReporter listener, which will write test results to an XML file, in a format similar to that used by the ant JUnit XML writer.
Let's specify the result_reporter in the configuration now. Edit the
…/res/config/example/helloworld.xml config like this:
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>
Now rebuild tradefed and re-run the hello world sample:
tf >run example/helloworld
05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt
05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0
Notice the log message stating that an XML file has been generated. The generated file should look like this:
<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
<properties />
<testcase name="sampleTest" classname="com.example.TestClassName" time="0">
<failure>oh noes, test failed
</failure>
</testcase>
</testsuite>
You can also write your own custom invocation listeners. It just needs to implement the ITestInvocationListener interface.
Also note that tradefed supports multiple invocation listeners, meaning that you can send test
results to multiple independent destinations. Just specify multiple
<result_reporter> tags in your config to do this.
TradeFederation includes two logging facilities:
Lets focus on #2 for now. Trade Federation's host logs are reported using the CLog wrapper for the ddmlib Log class.
Let's convert the previous System.out.println call in HelloWorldTest to a
CLog call:
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber());
Note that CLog handles string interpolation directly, akin to
String.format. At this point, when you rebuild and rerun TF, you should see the
log message on stdout.
tf> run example/helloworld
…
05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
…
By default, tradefed will
output host log messages to
stdout. TF also includes a log implementation that will write messages to a file:
FileLogger. To add file logging,
add a logger tag to the config, specifying the full class name of
FileLogger.
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
<logger class="com.android.tradefed.log.FileLogger" />
</configuration>
Now rebuild and run the helloworld example again.
tf >run example/helloworld
…
05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
Note the log message indicating the path of the host log. View the contents of that file, and you should see your HelloWorldTest log message
$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
The TradeFederation framework will also automatically capture the logcat from the allocated device,
and send it the invocation listener for processing. XmlResultReporter will save the
captured device logcat as a file.
Objects loaded from a Trade Federation Configuration (aka Configuration objects) also have the ability to receive data from command line arguments.
This is accomplished via the @Option annotation. To participate, a Configuration object class
would apply the @Option annotation to a member field, and provide it a unique name. This would
allow that member field's value to be populated via a command line option, and would also
automatically add that option to the configuration help system (Note: not all field types are
supported: see the
OptionSetter javadoc for a
description of supported types).
Let's add an @Option to the HelloWorldTest:
@Option(name="my_option",
shortName='m',
description="this is the option's help text",
// always display this option in the default help text
importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";
And let's add a log message to display the value of the option in HelloWorldTest, so we can demonstrate that it was received correctly.
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
…
CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption);
Rebuild TF and run helloworld; you should see a log message with my_option's
default value.
tf> run example/helloworld
…
05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault'
Now pass in a value for my_option: you should see my_option getting populated with that value
tf> run example/helloworld --my_option foo
…
05-24 18:33:44 I/HelloWorldTest: I received option 'foo'
TF configurations also include a help system, which automatically displays help text for
@Option fields. Try it now, and you should see the help text for
my_option:
tf> run --help example/helloworld
Printing help for only the important options. To see help for all options, use the --help-all flag
cmd_options options:
--[no-]help display the help text for the most important/critical options. Default: false.
--[no-]help-all display the full help text for all options. Default: false.
--[no-]loop keep running continuously. Default: false.
test options:
-m, --my_option this is the option's help text Default: thisisthedefault.
'file' logger options:
--log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.
Note the message at the top about "printing only the important options." To reduce option help
clutter, TF uses the Option#importance attribute to determine whether to show a
particular @Option field's help text
when --help is specified. --help-all will always show help for all
@Option fields, regardless of importance. See the
Option.Importance javadoc for details.
You can also specify an Option's value within the config by adding a
<option name="" value=""> element. Let's see how this looks in
helloworld.xml:
<test class="com.android.tradefed.example.HelloWorldTest" >
<option name="my_option" value="fromxml" />
</test>
Re-building and running helloworld should now produce this output:
05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml'
The configuration help should also be updated to indicate my_option's new default value:
tf> run --help example/helloworld
test options:
-m, --my_option this is the option's help text Default: fromxml.
Also note that other configuration objects included in the helloworld config, such as
FileLogger, also accept options. The option --log-level-display is
interesting because it filters the logs that show up on stdout. You may have noticed from earlier
in the tutorial that the "Hello, TF World! I have device …' log message stopped being displayed
on stdout once we switched to using FileLogger. You can increase the verbosity of
logging to stdout by passing in the --log-level-display arg.
Try this now, and you should see the 'I have device' log message reappear on stdout, in addition to being logged to a file.
tf >run --log-level-display info example/helloworld
…
05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
As a reminder, if you're stuck on something, the Trade Federation source code has a lot of useful information that isn't exposed in the documentation. And if all else fails, try asking on the android-platform Google Group, with "Trade Federation" in the message subject.