• Home
  • History
  • Annotate
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1
2Copyright (C) 2012 The Android Open Source Project
3
4Licensed under the Apache License, Version 2.0 (the "License");
5you may not use this file except in compliance with the License.
6You may obtain a copy of the License at
7
8     http://www.apache.org/licenses/LICENSE-2.0
9
10Unless required by applicable law or agreed to in writing, software
11distributed under the License is distributed on an "AS IS" BASIS,
12WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13See the License for the specific language governing permissions and
14limitations under the License.
15
16Intro
17-----
18We test shill using two sets of tests: unit tests, and integration
19tests. The unit tests are built using Google Test [1] and Google Mock
20[2]; the integration tests use autotest [3].
21
22Running unit tests for Chrome OS
23--------------------------------
24Here ${BOARD} is a valid board name, like link or x86-generic.
25- build the shill_unittest target
26- run the resulting shill_unittest binary
27- run the unit tests from your host machine under gdb
28  (chroot)$ FEATURES=test emerge-${BOARD} shill
29  (chroot)$ gdb_x86_local --board ${BOARD} \
30                /build/${BOARD}/var/cache/portage/chromeos-base/platform2/out/Default/shill_unittest
31  (Of course if the unit tests segfaulted, you wouldn't need the emerge
32  step since the build directory would have been retained in the course
33  of the test failing.)
34- The emerge workflow given above is incremental. It uses ninja to rebuild only
35  relevant objects in the shill target.
36- You can restrict the test runs to only shill unittests by using
37  (chroot)$ P2_TEST_FILTER="shill::*" FEATURES=test emerge-${BOARD} shill
38  The filter can be made more specific to include googletest filters like
39  "shill::CellularTest.StartGSMRegister"
40- if you want to set a breakpoint in gdb, make sure to include the shill
41  namespace. e.g., run
42    (cros-gdb) b shill::EthernetService::GetStorageIdentifier
43    Breakpoint 2 at 0x5555563cc270: file ethernet_service.cc, line 63.
44  rather than
45    (cros-gdb) b EthernetService::GetStorageIdentifier
46    Function "EthernetService::GetStorageIdentifier" not defined.
47    Make breakpoint pending on future shared library load? (y or [n]) n
48
49- alternate build command:
50  - Another way to build which uses the emerge command behind the scenes:
51    (chroot)$ cros_workon_make --board ${BOARD} shill
52  - to see the actual compiler commands that are run:
53    (chroot)$ CFLAGS="-print-cmdline" cros_workon_make --reconf \
54                --board=${BOARD} shill
55  - to abort compilation on the first error
56    (chroot)$ MAKEFLAGS="--stop" cros_workon_make --test --board=${BOARD} \
57                --reconf shill
58
59Running unit tests for Chrome OS with the address sanitizer
60-----------------------------------------------------------
61USE="asan clang wimax" TEST_FILTER="shill::*" emerge-${BOARD} shill
62
63This also turns on "wimax" and its tests, since this is disabled on most
64platforms.
65
66Running unit tests for Android
67------------------------------
68Currently, only native unit tests are supported on Android, meaning that unit tests
69can only be run on the target device.
70
71The test binary is located at /data/nativetest/shill_test/shill_test.
72By default, it does not have the execute permissions due to inherited
73directory security policy. Consequently, before running the unit tests,
74you will need to grant execute permissions to the test binary using
75the following command:
76  chmod +x /data/nativetest/shill_test/shill_test
77
78To run all unit tests:
79  /data/nativetest/shill_test/shill_test
80
81To run specific unit tests using filter:
82  /data/nativetest/shill_test/shill_test --gtest_filter=<TestClassName>.<TestName>
83
84For example:
85  /data/nativetest/shill_test/shill_test --gtest_filter=WiFiMainTest.OnNewWiphy
86  /data/nativetest/shill_test/shill_test --gtest_filter=WiFiMainTest.*
87
88Running integration tests
89-------------------------
90- build a test image, suitable for a VM:
91  (chroot) src/scripts$ ./build_packages --board=${BOARD}
92  (chroot) src/scripts$ ./build_image --board=${BOARD} \
93                            --noenable_rootfs_verification test
94  (chroot) src/scripts$ ./image_to_vm.sh --board=${BOARD} --test_image
95
96- start the VM
97  (host)$ sudo kvm -m 2048 -vga std -pidfile /tmp/kvm.pid \
98              -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \
99              -hda <path to chroot>/src/build/images/${BOARD}/latest/chromiumos_qemu_image.bin
100
101- DO NOT log in on the console.
102  (doing so will load a user profile onto shill's profile stack; this
103  interferes with the test profile that we use in the autotests)
104
105- if you've modified the source after building the image, update shill on
106  the image, and then restart shill:
107  (chroot) src/scripts$ ./start_devserver
108  (chroot) src/scripts$ ssh-keygen -R '[127.0.0.1]:9222'
109  (chroot) src/scripts$ emerge-${BOARD} platform2
110  (chroot) src/scripts$ cros deploy 127.0.0.1:9222 platform2
111
112  (chroot) src/scripts$ ssh -p 9222 root@127.0.0.1
113  localhost / # restart shill
114
115- run the tests
116  (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager
117                                  --args="config_file=wifi_vm_config"
118                                  --ssh_options="-p 9222"
119
120  To run a specific test out of the test suite, use test_pat option to --args.
121  # Example: To just run the 035CheckWEPKeySyntax test:
122  (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager
123                                  --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax"
124                                  --ssh_options="-p 9222"
125
126- configuration note: if you use a different port
127  (e.g. hostfwd=tcp::9223-:22), you'll need to change:
128    - the ssh_port argument to test_that
129    - the port numbers in
130      <chroot>/third_party/autotest/files/client/config/wifi_vm_config
131
132- debugging test failures
133  - "grep shill /var/log/messages" for log messages
134  - "grep wpa_supplicant /var/log/messages" for supplicant log messages
135  - "wpa_debug debug" to increase supplicant log level
136  - try resetting the test infrastructure
137    - rmmod mac80211_hwsim mac80211 cfg80211
138    - restart wpasupplicant
139    - rm /tmp/hostapd-test.control/*
140  - examine autotest log files
141    - check how far the test got before it failed
142      $ grep -a ': step ' <test output>/<suite name>/<suite name>.<test name>/debug/<suite name>.<test name>.INFO
143      e.g.
144      (chroot) $ grep -a ': step ' /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO
145    - read the log file
146      (chroot) $ LESSOPEN= less /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO
147
148      (LESSOPEN= prevents less from misinterpreting the logs as binary files,
149       and piping them through hexdump.)
150
151- additional test suites: we have a number of other WiFi test suites
152  (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf,
153  WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too
154  relevant to shill (just yet), but the rest probably are.
155
156[1] http://code.google.com/p/googletest/
157[2] http://code.google.com/p/googlemock/
158[3] http://autotest.kernel.org/,
159    http://www.chromium.org/chromium-os/testing/testing-faq
160