README.md
1# cli-test
2
3## What?
4
5`cli-test` makes integration testing of command-line tools easier.
6
7## Goals
8
9* Readable syntax. Common cases should be concise, and pretty much anyone
10 should be able to read tests even if they've never seen this tool before.
11
12* Minimal issues with quoting. The toybox tests -- being shell scripts --
13 quickly become a nightmare of quoting. Using a non ad hoc format (such as
14 JSON) would have introduced similar but different quoting issues. A custom
15 format, while annoying, side-steps this.
16
17* Sensible defaults. We expect your exit status to be 0 unless you say
18 otherwise. We expect nothing on stderr unless you say otherwise. And so on.
19
20* Convention over configuration. Related to sensible defaults, we don't let you
21 configure things that aren't absolutely necessary. So you can't keep your test
22 data anywhere except in the `files/` subdirectory of the directory containing
23 your test, for example.
24
25## Non Goals
26
27* Portability. Just being able to run on Linux (host and device) is sufficient
28 for our needs. macOS is probably easy enough if we ever need it, but Windows
29 probably doesn't make sense.
30
31## Syntax
32
33Any all-whitespace line, or line starting with `#` is ignored.
34
35A test looks like this:
36```
37name: unzip -l
38command: unzip -l $FILES/example.zip d1/d2/x.txt
39after: [ ! -f d1/d2/x.txt ]
40expected-stdout:
41 Archive: $FILES/example.zip
42 Length Date Time Name
43 --------- ---------- ----- ----
44 1024 2017-06-04 08:45 d1/d2/x.txt
45 --------- -------
46 1024 1 file
47---
48```
49
50The `name:` line names the test, and is only for human consumption.
51
52The `command:` line is the command to be run. Additional commands can be
53supplied as zero or more `before:` lines (run before `command:`) and zero or
54more `after:` lines (run after `command:`). These are useful for both
55setup/teardown but also for testing post conditions (as in the example above).
56
57Any `command:`, `before:`, or `after:` line is expected to exit with status 0.
58Anything else is considered a test failure.
59
60The `expected-stdout:` line is followed by zero or more tab-prefixed lines that
61are otherwise the exact output expected from the command. (There's magic behind
62the scenes to rewrite the test files directory to `$FILES` because otherwise any
63path in the output would depend on the temporary directory used to run the test.)
64
65There is currently no `expected-stderr:` line. Standard error is implicitly
66expected to be empty, and any output will cause a test failure. (The support is
67there, but not wired up because we haven't needed it yet.)
68
69The fields can appear in any order, but every test must contain at least a
70`name:` line and a `command:` line.
71
72## Output
73
74The output is intended to resemble gtest.
75
76## Future Directions
77
78* It's often useful to be able to *match* against stdout/stderr/a file rather
79 than give exact expected output. We might want to add explicit support for
80 this. In the meantime, it's possible to use an `after:` with `grep -q` if
81 you redirect in your `command:`.
82
83* In addition to using a `before:` (which will fail a test), it can be useful
84 to be able to specify tests that would cause us to *skip* a test. An example
85 would be "am I running as root?".
86
87* It might be useful to be able to make exit status assertions other than 0?
88
89* There's currently no way (other than the `files/` directory) to share repeated
90 setup between tests.
91