1# Setup: Ubuntu host, QEMU vm, x86-64 kernel
2
3These are the instructions on how to fuzz the x86-64 kernel in a QEMU with Ubuntu 14.04 on the host machine and Debian Wheezy in the QEMU instances.
4
5## GCC
6
7Since syzkaller requires coverage support in GCC, we need to use a recent GCC version. To checkout GCC 7.1.0 sources to `$GCC` dir:
8``` bash
9svn checkout svn://gcc.gnu.org/svn/gcc/trunk $GCC
10cd $GCC
11svn ls -v ^/tags | grep gcc_7_1_0_release
12svn up -r 247494
13```
14
15Unfortunately there's a typo in the source of `gcc_7_1_0_release`. Apply [this fix](https://patchwork.ozlabs.org/patch/757421/):
16``` c
17diff --git a/gcc/tree.h b/gcc/tree.h
18index 3bca90a..fdaa7af 100644
19--- a/gcc/tree.h
20+++ b/gcc/tree.h
21@@ -897,8 +897,8 @@  extern void omp_clause_range_check_failed (const_tree, const char *, int,
22 /* If this is true, we should insert a __cilk_detach call just before
23    this function call.  */
24 #define EXPR_CILK_SPAWN(NODE) \
25-  (tree_check2 (NODE, __FILE__, __LINE__, __FUNCTION__, \
26-                CALL_EXPR, AGGR_INIT_EXPR)->base.u.bits.unsigned_flag)
27+  (TREE_CHECK2 (NODE, CALL_EXPR, \
28+                AGGR_INIT_EXPR)->base.u.bits.unsigned_flag)
29
30 /* In a RESULT_DECL, PARM_DECL and VAR_DECL, means that it is
31    passed by invisible reference (and the TREE_TYPE is a pointer to the true
32```
33
34Install GCC prerequisites:
35```
36sudo apt-get install flex bison libc6-dev libc6-dev-i386 linux-libc-dev linux-libc-dev:i386 libgmp3-dev libmpfr-dev libmpc-dev build-essential bc
37```
38
39Build GCC:
40``` bash
41mkdir build
42mkdir install
43cd build/
44../configure --enable-languages=c,c++ --disable-bootstrap --enable-checking=no --with-gnu-as --with-gnu-ld --with-ld=/usr/bin/ld.bfd --disable-multilib --prefix=$GCC/install/
45make -j64
46make install
47```
48
49Now you should have GCC binaries in `$GCC/install/bin/`:
50``` bash
51$ ls $GCC/install/bin/
52c++  gcc-ar      gcov-tool                x86_64-pc-linux-gnu-gcc-7.0.0
53cpp  gcc-nm      x86_64-pc-linux-gnu-c++  x86_64-pc-linux-gnu-gcc-ar
54g++  gcc-ranlib  x86_64-pc-linux-gnu-g++  x86_64-pc-linux-gnu-gcc-nm
55gcc  gcov        x86_64-pc-linux-gnu-gcc  x86_64-pc-linux-gnu-gcc-ranlib
56```
57
58## Kernel
59
60Checkout Linux kernel source:
61``` bash
62git clone https://github.com/torvalds/linux.git $KERNEL
63```
64
65Generate default configs:
66``` bash
67cd $KERNEL
68make defconfig
69make kvmconfig
70```
71
72Now we need to enable some config options required for syzkaller.
73Edit `.config` file manually and enable:
74```
75CONFIG_KCOV=y
76CONFIG_DEBUG_INFO=y
77CONFIG_KASAN=y
78CONFIG_KASAN_INLINE=y
79```
80
81You might also want to enable some other kernel configs as described [here](kernel_configs.md).
82
83Since enabling these options results in more sub options being available, we need to regenerate config. Run this and press enter each time when prompted for some config value to leave it as default:
84``` bash
85make oldconfig
86```
87
88Build the kernel with previously built GCC:
89```
90make CC="$GCC/install/bin/gcc" -j64
91```
92
93Now you should have `vmlinux` (kernel binary) and `bzImage` (packed kernel image):
94``` bash
95$ ls $KERNEL/vmlinux
96$KERNEL/vmlinux
97$ ls $KERNEL/arch/x86/boot/bzImage
98$KERNEL/arch/x86/boot/bzImage
99```
100
101## Image
102
103Install debootstrap:
104``` bash
105sudo apt-get install debootstrap
106```
107
108Use [this script](https://github.com/google/syzkaller/blob/master/tools/create-image.sh) to create a minimal Debian-wheezy Linux image.
109The result should be `$IMAGE/wheezy.img` disk image.
110
111Sometimes it's useful to have some additional packages and tools available in the VM even though they are not required to run syzkaller.
112The instructions to install some useful tools are below.
113They should obviously be executed before packing the `.img` file.
114
115To install other packages (not required to run syzkaller):
116``` bash
117sudo chroot wheezy /bin/bash -c "apt-get update; apt-get install -y curl tar time strace gcc make sysbench git vim screen usbutils"
118```
119
120To install Trinity (not required to run syzkaller):
121``` bash
122sudo chroot wheezy /bin/bash -c "mkdir -p ~; cd ~/; wget https://github.com/kernelslacker/trinity/archive/v1.5.tar.gz -O trinity-1.5.tar.gz; tar -xf trinity-1.5.tar.gz"
123sudo chroot wheezy /bin/bash -c "cd ~/trinity-1.5 ; ./configure.sh ; make -j16 ; make install"
124```
125
126To install perf (not required to run syzkaller):
127``` bash
128cp -r $KERNEL wheezy/tmp/
129sudo chroot wheezy /bin/bash -c "apt-get update; apt-get install -y flex bison python-dev libelf-dev libunwind7-dev libaudit-dev libslang2-dev libperl-dev binutils-dev liblzma-dev libnuma-dev"
130sudo chroot wheezy /bin/bash -c "cd /tmp/linux/tools/perf/; make"
131sudo chroot wheezy /bin/bash -c "cp /tmp/linux/tools/perf/perf /usr/bin/"
132rm -r wheezy/tmp/linux
133```
134
135## QEMU
136
137Install `QEMU`:
138``` bash
139sudo apt-get install kvm qemu-kvm
140```
141
142Make sure the kernel boots and `sshd` starts:
143``` bash
144qemu-system-x86_64 \
145  -kernel $KERNEL/arch/x86/boot/bzImage \
146  -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
147  -hda $IMAGE/wheezy.img \
148  -net user,hostfwd=tcp::10021-:22 -net nic \
149  -enable-kvm \
150  -nographic \
151  -m 2G \
152  -smp 2 \
153  -pidfile vm.pid \
154  2>&1 | tee vm.log
155```
156
157```
158early console in setup code
159early console in extract_kernel
160input_data: 0x0000000005d9e276
161input_len: 0x0000000001da5af3
162output: 0x0000000001000000
163output_len: 0x00000000058799f8
164kernel_total_size: 0x0000000006b63000
165
166Decompressing Linux... Parsing ELF... done.
167Booting the kernel.
168[    0.000000] Linux version 4.12.0-rc3+ ...
169[    0.000000] Command line: console=ttyS0 root=/dev/sda debug earlyprintk=serial
170...
171[ ok ] Starting enhanced syslogd: rsyslogd.
172[ ok ] Starting periodic command scheduler: cron.
173[ ok ] Starting OpenBSD Secure Shell server: sshd.
174```
175
176After that you should be able to ssh to QEMU instance in another terminal:
177``` bash
178ssh -i $IMAGE/ssh/id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhost
179```
180
181If this fails with "too many tries", ssh may be passing default keys before
182the one explicitly passed with `-i`. Append option `-o "IdentitiesOnly yes"`.
183
184To kill the running QEMU instance:
185``` bash
186kill $(cat vm.pid)
187```
188
189## Go
190
191Install Go 1.8.1:
192``` bash
193wget https://storage.googleapis.com/golang/go1.8.1.linux-amd64.tar.gz
194tar -xf go1.8.1.linux-amd64.tar.gz
195mv go goroot
196export GOROOT=`pwd`/goroot
197export PATH=$GOROOT/bin:$PATH
198mkdir gopath
199export GOPATH=`pwd`/gopath
200```
201
202## syzkaller
203
204Get and build syzkaller:
205``` bash
206go get -u -d github.com/google/syzkaller/...
207cd gopath/src/github.com/google/syzkaller/
208mkdir workdir
209make
210```
211
212Create a manager config like the following, replacing the environment
213variables `$GOPATH`, `$KERNEL` and `$IMAGE` with their actual values.
214```
215{
216	"target": "linux/amd64",
217	"http": "127.0.0.1:56741",
218	"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir",
219	"kernel_obj": "$KERNEL",
220	"image": "$IMAGE/wheezy.img",
221	"sshkey": "$IMAGE/ssh/id_rsa",
222	"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
223	"procs": 8,
224	"type": "qemu",
225	"vm": {
226		"count": 4,
227		"kernel": "$KERNEL/arch/x86/boot/bzImage",
228		"cpu": 2,
229		"mem": 2048
230	}
231}
232```
233
234Run syzkaller manager:
235``` bash
236./bin/syz-manager -config=my.cfg
237```
238
239Now syzkaller should be running, you can check manager status with your web browser at `127.0.0.1:56741`.
240
241If you get issues after `syz-manager` starts, consider running it with the `-debug` flag.
242Also see [this page](/docs/troubleshooting.md) for troubleshooting tips.
243