1<a name="faft" /> 2 3# [FAFT (Fully Automated Firmware Test) 4 5_Last updated: 2011/11/08_ 6 7_Last content refresh from go/faft: 2019/02/19_ 8 9- [FAFT (Fully Automated Firmware Test)](#faft) 10 - [FAFT Related Documents](#faft-related-documents) 11 - [Introduction](#introduction) 12 - [Backgrounds](#backgrounds) 13 - [State of the Art](#state-of-the-art) 14 - [VBoot_Reference Library Tests](#vboot-reference-library-tests) 15 - [U-Boot vbexport/vboot Tests](#u-boot-vbexport-vboot-tests) 16 - [SAFT (Semi Automated Firmware Test)](#saft-semi-automated-firmware-test) 17 - [Factory Regression Testing Using Servo](#factory-regression-testing-using-servo) 18 - [U-Boot Test System ](#u-boot-test-system) 19 - [FAFT Implementation Details](#faft-implementation-details) 20 - [Test Environment](#test-environment) 21 - [Test Harness](#test-harness) 22 - [Software Architecture](#software-architecture) 23 - [Test Case Example](#test-case-example) 24 - [FAFT Scopes](#faft-scopes) 25 - [Scope 1: Enhance SAFT in Automated Way on Autotest Harness](#scope-1) 26 - [Scope 2: Test the Layer of VBExport APIs](#scope-2) 27 - [Scope 3: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows](#scope-3) 28 - [Scope 4: Monkey Test during Firmware/EC update](#scope-4) 29 - [Other Ideas](#other-ideas) 30 31<a name="faft-related-documents" /> 32 33## FAFT Related Documents 34 35- FAFT Setup Instructions: [http://goto/faft-setup](http://goto/faft-setup) 36- FAFT Test Coverage: [http://goto/faft-coverage](http://goto/faft-coverage) 37- FAFT Comparing Existing Manual Test Cases: [http://goto/faft-comparison](http://goto/faft-comparison) 38 39<a name="introduction" /> 40 41## Introduction 42 43Security is one of the selling points of Chrome OS netbooks. Verified boot provides a solution to Chrome OS security model. In the verified boot design, firmware is the foundation of this secured castle. In addition, part of the firmware is marked as read-only, that means we are unable to patch security holes via autoupdate. So firmware testing is very important; however, our current firmware is lack of automated tests. It only relies on the manual tests by developers, our test team, and dogfooders. It seems to be a risk. 44 45This document proposes a solution to the test problem. The advantages of FAFT: 46- Fully automatic, no human involved; 47- Real test of physical hardware, like USB plug-in, Ctrl-D key press; 48- High test coverage of complicated verified boot flows; 49- Easy to integrate to our existing test infrastructure, i.e. deploy to test lab, run remotely, continue test, etc. 50 51<a name="backgrounds" /> 52 53## Backgrounds 54 55This document uses [U-Boot](http://en.wikipedia.org/wiki/Das_U-Boot) as an example to discuss the FAFT test approach. It can be also applied to [coreboot](http://en.wikipedia.org/wiki/Coreboot) and any proprietary BIOS in a similar way. 56 57The U-Boot software stack looks like the following graph: [cl/70339149](https://critique.corp.google.com/#review/70339149) 58 59<img src="assets/faft-u-boot-stack.png" /> 60 61Most of the verified boot logic and crypto algorithms are inside the vboot_reference library. The main control logic in U-Boot is simple, just like reading some headers from SPI and filling them in a data structure which is then passed to vboot_reference. 62 63The underlying U-Boot drivers are abstracted into VBoot Export APIs and some (not too many) APIs which the main control logic is used. VBoot export APIs are the most, e.g., TPM, display, keyboard, disk, system related, etc. The main control logic used APIs are not many, e.g., SPI, GPIO, etc. The latter APIs are not included in VBoot Export APIs because some x86 legacy reasons. They may be merged in the future. 64 65The firmware boot path is like the following graph: 66 67<img src="assets/faft-firmware-boot-path.png" /> 68 69Our tests are mostly focused on VBoot Bootstub and VBoot Main Firmware since they are complicated and many decision branches. U-Boot initialization and kernel booting are simple and single path decisions. So full system regression tests are enough. 70 71<a name="state-of-the-art" /> 72 73## State of the Art 74 75<a name="vboot-reference-library-tests" /> 76 77### VBoot_Reference Library Tests 78 79 80Source: src/platform/vboot_reference/tests 81 82vboot_reference provides a lot of tests varying from crypto algorithms to vboot main logic and control flows. These tests stub the underlying APIs as all success functions. So it is unable to validate the correctness of underlying BIOS platform and the hardware behaviors. 83 84<img src="assets/faft-vboot-reference-tests.png" /> 85 86<a name="u-boot-vbexport-vboot-tests" /> 87 88### U-Boot vbexport/vboot Tests 89 90Source: src/third_party/u-boot/files/common/cmd_vb*_test.c 91 92Almost all the U-Boot drivers are abstracted into couple of major APIs: VBExport APIs and main logic used driver APIs. U-Boot provides a CLI for manual diagnostic. We wrote some commands to test these APIs. For example, for testing the disk read/write functions, a test command is to write some patterns to the disk and read them back to verify. However, some of these tests needs human involved, like inserting an USB stick to check if it is detectable. 93 94<img src="assets/faft-u-boot-vbexport-vboot-test.png" /> 95 96<a name="saft-semi-automated-firmware-test" /> 97 98### SAFT (Semi Automated Firmware Test) 99 100Source: src/platform/saft 101 102First of all, SAFT only works on x86 and hasn’t ported to ARM yet. 103 104SAFT assumes the whole firmware as a black-box and tests the whole system from firmware, kernel, to rootfs. For example, it corrupts firmware RW A/B to see if it enters recovery mode success. However, SAFT also needs human involved in many cases. 105 106<img src="assets/faft-saft.png" /> 107 108<a name="factory-regression-testing-using-servo" /> 109 110### Factory Regression Testing Using Servo 111 112Design Doc: [https://docs.google.com/a/google.com/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US](https://docs.google.com/a/google.com/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US) 113 114This work is an on-going project to ensure the whole factory install flow work fine. It downloads a factory bundle from buildbot, setup TFTP and Omaha servers. It uses a mini-Servo board to control a Chrome OS device, like switching dev switch, resetting device, sending Ctrl-D key, etc. This work is done by Rong Chang and the factory team to ensure partners get a good factory bundle. 115 116Since using a Servo board can automate almost all the human behaviors. The test can be done fully automated. The proposed FAFT in this document is an extension of this work. 117 118<a name="u-boot-test-system" /> 119 120### U-Boot Test System 121 122Design Doc: [https://docs.google.com/a/google.com/document/d/1JorndqNOrEOXgdMRuNBPk1o73XKl1JwCwLkDSOQy7wU/edit?hl=en_US](https://docs.google.com/a/google.com/document/d/1JorndqNOrEOXgdMRuNBPk1o73XKl1JwCwLkDSOQy7wU/edit?hl=en_US) 123 124Simon proposed a U-Boot native test system to split the U-Boot components into platform code and generic U-Boot code + Verified Bood code. By mocking the platform code. we can test the functions of VBoot library and it integration in a platform-independent way. On the other side, the architecture-specific platform code can be tested in a separated way. 125 126<a name="faft-implementation-details" /> 127 128## FAFT Implementation Details 129 130<a name="test-environment" /> 131 132### Test Environment 133 134We need a [Servo board](https://sites.google.com/a/google.com/chromeos-partner/hardware-control-and-debug/servo) to connect a DUT such that we can: 135 136- flash a new firmware image; 137- run regression tests; 138- simulate lid close, power cycles, gpios, and key presses; 139- get console output via UART; 140- simulate USB plug in/out, by either muxing a physical USB stick or emulating a USB storage gadget in the host machine. 141 142<img src="assets/faft-test-environment.png" /> 143 144<a name="test-harness" /> 145 146### Test Harness 147 148We use Autotest as our test harness and run FAFT as server tests. The host uses Ethernet to connect a DUT to push the test commands to it. Since Autotest is widely used in the Chrome OS projects. We can benefit in: 149 150- easy to integrate to buildbot; 151- easy to run remotely; 152- easy to deploy to the test lab. 153 154Some existing works using Servo board test on top on Autotest can be found in: 155 156- src/third_party/autotest/files/server/site_tests/platform_CloseOpenLid 157- src/third_party/autotest/files/server/site_tests/platform_LongPressPower 158 159FAFT is also based on the existing ServoTest approach and does some enhancements. 160 161<a name="software-architecture" /> 162 163### Software Architecture 164 165Here is our proposed FAFT framework. The red pieces are missing and need to be implemented in this project. 166 167<img src="assets/faft-software-architecture.png" /> 168 169All the FAFT test cases are server side tests, meaning that all the actions are controlled by Host. It is flexible to keep states after sequential reboot. However, we still want to get the DUT state and run some command on it, like getting current firmware version, setting the fw_try_b flag, etc. These actions should be completed in the DUT. So we setup a FAFT Client for this purpose and use RPC to communicate with it. 170 171<a name="test-case-example" /> 172 173### Test Case Example 174 175Here is an example to demonstrate how easy it is to write a FAFT test case. It test the firmware like: 1761. FW A normal boot -> enable devsw + set try_fwb -> reboot -> press ctrl-d 1772. FW B developer boot -> do nothing -> reboot -> press ctrl-d 1783. FW A developer boot -> disable devsw -> reboot -> do nothing 1794. FW A normal boot -> finished 180 181``` 182class firmware_DevModeTryB(FAFTSequence): 183 def run_once(self, host=None): 184 self.register_faft_sequence(( 185 # ------------------------- step 1 ------------------------- 186 {‘state_checker’: (self.faft_util.crossystem_checker, 187 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}), 188 ‘userspace_action’: self.enable_devmode_and_set_try_fwb, 189 ‘firmware_action’: self.wait_and_press_ctrl_d}, 190 # ------------------------- step 2 ------------------------- 191 {‘state_checker’, (self.faft_util.crossystem_checker, 192 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘B’}), 193 ‘userspace_action’: None, 194 ‘firmware_action’: self.wait_and_press_ctrl_d}, 195 # ------------------------- step 3 ------------------------- 196 {‘state_checker’, (self.faft_util.crossystem_checker, 197 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘A’}), 198 ‘userspace_action’: self.servo.disable_development_mode, 199 ‘firmware_action’: None}, 200 # ------------------------- step 4 ------------------------- 201 {‘state_checker’, (self.faft_util.crossystem_checker, 202 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}), 203 } 204 )) 205 self.run_faft_sequence() 206 207 def enable_devmode_and_set_try_fwb(self): 208 self.faft_client.crossystem.set_try_fwb() 209 self.servo.enable_development_mode() 210 211 def wait_and_press_ctrl_d(self): 212 time.sleep(10) 213 self.servo.ctrl_d() 214``` 215 216<a name="faft-scopes" /> 217 218## FAFT Scopes 219 220There are several scopes about FAFT. 221 222<a name="scope-1" /> 223 224### Scope 1: Enhance SAFT in Automated Way on Autotest Harness 225 226SAFT provides a complete set of libraries to access system info and corrupt/restore firmware and kernel. It also has a set of test cases and expected results. For example, in the SAFT main program, `saft_utility.py`: 227 228``` 229TEST_STATE_SEQUENCE = ( 230 ('1:1:1:0:3', FST.set_try_fw_b), # Step 0 231 ('1:2:1:0:3', None), 232 ('1:1:1:0:3', FST.corrupt_firmware, 'a'), 233 ('1:2:1:0:3', FST.restore_firmware, 'a'), 234 ('1:1:1:0:3', FST.corrupt_firmware, ('a', 'b')), 235 ('5:0:1:1:3', FST.restore_firmware, ('a', 'b')), # Step 5 236 ('1:1:1:0:3', FST.corrupt_kernel, 'a'), 237 ('1:1:1:0:5', FST.corrupt_kernel, 'b'), 238 ('6:0:1:1:3', FST.restore_kernel, ('a', 'b')), 239 ('1:1:1:0:3', FST.request_recovery_boot), 240 ('8:0:1:1:3', FST.prepare_tpm_tests), # Step 10 241 ('1:1:1:0:5', FST.move_kernel_backward, 'b'), 242 ('6:0:1:1:3', FST.jump_kernels_forward), 243 ('1:1:1:0:3', FST.move_firmware_backward, 'a'), 244 ('1:2:1:0:3', FST.move_firmware_backward, 'b'), 245 ('5:0:1:1:3', FST.jump_firmwares_forward), # Step 15 246 ('1:1:1:0:3', FST.request_recovery_boot), 247 ('8:0:1:1:3', FST.terminate_tpm_tests), 248 ('*:*:*:*:*', cgpt_st.test_loop), 249 ('1:1:1:0:3', FST.revert_firmware), 250 ('1:1:1:0:3', None), # Step 20 251 ) 252``` 253 254It is good that we don’t need to reimplement the same work. FAFT still uses the original SAFT test cases and libraries in the first step and port them to Autotest harness. FAFT does a bit modification to makes all test cases automated by using the powerful Servo board functions. 255 256Current SAFT is a sequence of actions and expected results to test all firmware / kernel / recovery / tpm behaviors as a whole. SAFT does tests in this way because we want to simply the human actions, not easy to be confused. Since FAFT can simulate all human actions. We can separate the test sequence more isolated and one for a special test purpose. 257 258A demo of porting SAFT test cases on FAFT: [https://video.google.com/a/?pli=1#/Play/contentId=60bdda4552e4c6a2](https://video.google.com/a/?pli=1#/Play/contentId=60bdda4552e4c6a2) 259 260This demo presents the feasibility of the integration of SAFT library and FAFT framework. We still have a lot of work to make all the features completed. The tasks like: 261- Create a separated FAFT client and wrap all existing SAFT libraries into it; 262- Setup RPC to invoke FAFT client instead of using SSH tunnel like the demo; 263- Enhance the functions of FAFT server side utilities; 264- Make FAFT framework more generalized to fit any possible test cases; 265- Keep the state and backup firmware/kernel in the server side instead of USB; 266- Automatic setup DUT disk and USB image when given a buildbot image. 267 268<a name="scope-2" /> 269 270### Scope 2: Test the Layer of VBExport APIs 271 272In order to increase the test coverage and verify the underlying BIOS functions and drivers, a good entry is to re-run all existing vbexport / vboot tests in an automated way using our test framework!. See more detail in the State of the Art Section. We can catch bugs easily if any drivers changed or the BIOS source rebased. 273 274<img src="assets/faft-u-boot-vbexport-vboot-test.png" /> 275 276<a name="scope-3" /> 277 278### Scope 3: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows 279 280The normal boot and developer boot can be verified by SAFT. However, we still missed the test of the whole Recovery process. The whole recovery sequence is complicated: 2811. DD the recovery shim image to a USB stick; 2822. Hold Recovery button and power on DUT; 2833. Press SPACE; 2844. Insert a recovery shim; 2855. Wait for the complete of recovery process; 2866. Reboot; 2877. Boot to normal Chrome OS. 288 289This sequence can be fully automated by FAFT. For example, in step 1, we require an USB stick plugged to the Servo board by default. We first mux it to connect to the host machine such that we can write the recovery shim image to it. In step 4, we then mux the USB stick to the DUT. Another approach is to emulate an USB storage gadget in the host machine, it would be more flexible to control its functionalities by software. 290 291Moreover, there are many recovery mode variants, like normal recovery/devloper recovery, physical recovery button/keyboard simulated button, etc. It’d be easy to verify all variants by writing different test cases. 292 293Besides, factory install shim, firmware AU, reset shim, RMA, etc, are more complicated and have many variants. Factory install process currently will be tested by an automated script written by the factory team. Firmware AU still relies on manual tests, see the [test matrix](https://docs.google.com/a/google.com/spreadsheet/ccc?key=0AoHv3bO2oLIjdGQ5cnJ6eWc1ZnpjYzlaWFA4T0xlVVE&hl=en_US&authkey=CJWF8-gP#gid=8). Reset shim and RMA has no test yet. Actually, all these complicated flows can be tested in FAFT. 294 295<a name="scope-4" /> 296 297### Scope 4: Monkey Test during Firmware/EC update 298 299Firmware/EC update is a risky thing and may brick the machine. For example, the bug [http://crosbug.com/p/4419](http://crosbug.com/p/4419) reported, during the EC update, a user presses Power Button at the same time that bricks the machine. It is because the powerd triggers the shutdown process immediately and the system goes to off, but it cannot be turned on again by power button (until a cold boot). Note that the EC is still in update mode now. 300 301 302To catch bugs like that, we need a monkey test during firmware/EC update, like randomly switching GPIO, pressing Power Button, closing LID, pressing keyboard, plugging USB/SD disk, etc. 303 304<a name="other-ideas" /> 305 306## Other Ideas 307 308- Run FAFT on every build, integrated with buildbot 309- Stress test to ensure the stability of hardware and drivers 310- Cold reboot test 311- Use HDMI capture device or camera + image recognition to verify the display output 312
