1Some examples for inject
2
3inject guarantees the appropriate erroneous return of the specified injection
4mode (kmalloc,bio,etc) given a call chain and an optional set of predicates. You
5can also optionally print out the generated BPF program for
6modification/debugging purposes.
7
8As a simple example, let's say you wanted to fail all mounts. As of 4.17 we can
9fail syscalls directly, so let's do that:
10
11# ./inject.py kmalloc -v 'SyS_mount()'
12
13The first argument indicates the mode (or what to fail). Appropriate headers are
14specified, if necessary. The verbosity flag prints the generated program. Note
15that some syscalls will be available as 'SyS_xyz' and some will be available as
16'sys_xyz'. This is largely dependent on the number of arguments each syscall
17takes.
18
19Trying to mount various filesystems will fail and report an inability to
20allocate memory, as expected.
21
22Whenever a predicate is missing, an implicit "(true)" is inserted. The example
23above can be explicitly written as:
24
25# ./inject.py kmalloc -v '(true) => SyS_mount()(true)'
26
27The "(true)" without an associated function is a predicate for the error
28injection mechanism of the current mode. In the case of kmalloc, the predicate
29would have access to the arguments of:
30
31	int should_failslab(struct kmem_cache *s, gfp_t gfpflags);
32
33The bio mode works similarly, with access to the arguments of:
34
35	static noinline int should_fail_bio(struct bio *bio)
36
37We also note that it's unnecessary to state the arguments of the function if you
38have no intention to reference them in the associated predicate.
39
40Now let's say we want to be a bit more specific; suppose you want to fail
41kmalloc() from mount_subtree() when called from btrfs_mount(). This will fail
42only btrfs mounts:
43
44# ./inject.py kmalloc -v 'mount_subtree() => btrfs_mount()'
45
46Attempting to mount btrfs filesystem during the execution of this command will
47yield an error, but other filesystems will be fine.
48
49Next, lets say we want to hit one of the BUG_ONs in fs/btrfs. As of 4.16-rc3,
50there is a BUG_ON in btrfs_prepare_close_one_device() at fs/btrfs/volumes.c:1002
51
52To hit this, we can use the following:
53
54# ./inject.py kmalloc -v 'btrfs_alloc_device() => btrfs_close_devices()'
55
56While the script was executing, I mounted and unmounted btrfs, causing a
57segfault on umount(since that satisfied the call path indicated). A look at
58dmesg will confirm that the erroneous return value injected by the script
59tripped the BUG_ON, causing a segfault down the line.
60
61In general, it's worth noting that the required specificity of the call chain is
62dependent on how much granularity you need. The example above might have
63performed as expected without the intermediate btrfs_alloc_device, but might
64have also done something unexpected(an earlier kmalloc could have failed before
65the one we were targetting).
66
67For hot paths, the approach outlined above isn't enough. If a path is traversed
68very often, we can distinguish distinct calls with function arguments. Let's say
69we want to fail the dentry allocation of a file creatively named 'bananas'. We
70can do the following:
71
72# ./inject.py kmalloc -v 'd_alloc_parallel(struct dentry *parent, const struct
73qstr *name)(STRCMP(name->name, 'bananas'))'
74
75While this script is executing, any operation that would cause a dentry
76allocation where the name is 'bananas' fails, as expected.
77
78Here, since we're referencing a function argument in our predicate, we need to
79provide the function signature up to the argument we're using.
80
81To note, STRCMP is a workaround for some rewriter issues. It will take input of
82the form (x->...->z, 'literal'), and generate some equivalent code that the
83verifier is more friendly about. It's not horribly robust, but works for the
84purposes of making string comparisons a bit easier.
85
86Finally, we briefly demonstrate how to inject bio failures. The mechanism is
87identical, so any information from above will apply.
88
89Let's say we want to fail bio requests when the request is to some specific
90sector. An example use case would be to fail superblock writes in btrfs. For
91btrfs, we know that there must be a superblock at 65536 bytes, or sector 128.
92This allows us to run the following:
93
94# ./inject.py bio -v -I 'linux/blkdev.h'  '(({struct gendisk *d = bio->bi_disk;
95struct disk_part_tbl *tbl = d->part_tbl; struct hd_struct **parts = (void *)tbl +
96sizeof(struct disk_part_tbl); struct hd_struct **partp = parts + bio->bi_partno;
97struct hd_struct *p = *partp; dev_t disk = p->__dev.devt; disk ==
98MKDEV(254,16);}) && bio->bi_iter.bi_sector == 128)'
99
100The predicate in the command above has two parts. The first is a compound
101statement which shortens to "only if the system is btrfs", but is long due
102to rewriter/verifier shenanigans. The major/minor information can be found
103however; I used Python. The second part simply checks the starting
104address of bi_iter. While executing, this script effectively fails superblock
105writes to the superblock at sector 128 without affecting other filesystems.
106
107As an extension to the above, one could easily fail all btrfs superblock writes
108(we only fail the primary) by calculating the sector number of the mirrors and
109amending the predicate accordingly.
110
111Inject also provides a probability option; this allows you to fail the
112path+predicates some percentage of the time. For example, let's say we want to
113fail our mounts half the time:
114
115# ./inject.py kmalloc -v -P 0.01 'SyS_mount()'
116
117USAGE message:
118usage: inject.py [-h] [-I header] [-P probability] [-v] {kmalloc,bio} spec
119
120Fail specified kernel functionality when call chain and predicates are met
121
122positional arguments:
123  {kmalloc,bio}         indicate which base kernel function to fail
124  spec                  specify call chain
125
126optional arguments:
127  -h, --help            show this help message and exit
128  -I header, --include header
129                        additional header files to include in the BPF program
130  -P probability, --probability probability
131                        probability that this call chain will fail
132  -v, --verbose         print BPF program
133
134EXAMPLES:
135# ./inject.py kmalloc -v 'SyS_mount()'
136    Fails all calls to syscall mount
137# ./inject.py kmalloc -v '(true) => SyS_mount()(true)'
138    Explicit rewriting of above
139# ./inject.py kmalloc -v 'mount_subtree() => btrfs_mount()'
140    Fails btrfs mounts only
141# ./inject.py kmalloc -v 'd_alloc_parallel(struct dentry *parent, const struct \
142    qstr *name)(STRCMP(name->name, 'bananas'))'
143    Fails dentry allocations of files named 'bananas'
144# ./inject.py kmalloc -v -P 0.01 'SyS_mount()'
145    Fails calls to syscall mount with 1% probability
146