1Demonstrations of filetop, the Linux eBPF/bcc version. 2 3 4filetop shows reads and writes by file, with process details. For example: 5 6# ./filetop -C 7Tracing... Output every 1 secs. Hit Ctrl-C to end 8 908:00:23 loadavg: 0.91 0.33 0.23 3/286 26635 10 11PID COMM READS WRITES R_Kb W_Kb T FILE 1226628 ld 161 186 643 152 R built-in.o 1326634 cc1 1 0 200 0 R autoconf.h 1426618 cc1 1 0 200 0 R autoconf.h 1526634 cc1 12 0 192 0 R tracepoint.h 1626584 cc1 2 0 143 0 R mm.h 1726634 cc1 2 0 143 0 R mm.h 1826631 make 34 0 136 0 R auto.conf 1926634 cc1 1 0 98 0 R fs.h 2026584 cc1 1 0 98 0 R fs.h 2126634 cc1 1 0 91 0 R sched.h 2226634 cc1 1 0 78 0 R printk.c 2326634 cc1 3 0 73 0 R mmzone.h 2426628 ld 18 0 72 0 R hibernate.o 2526628 ld 16 0 64 0 R suspend.o 2626628 ld 16 0 64 0 R snapshot.o 2726628 ld 16 0 64 0 R qos.o 2826628 ld 13 0 52 0 R main.o 2926628 ld 12 0 52 0 R swap.o 30[...] 31 32This shows various files read and written during a Linux kernel build. By 33default the output is sorted by the total read size in Kbytes (R_Kb). Sorting 34order can be changed via -s option. This is instrumenting at the VFS interface, 35so this is reads and writes that may return entirely from the file system cache 36(page cache). 37 38While not printed, the average read and write size can be calculated by 39dividing R_Kb by READS, and the same for writes. 40 41The "T" column indicates the type of the file: "R" for regular files, "S" for 42sockets, and "O" for other (including pipes). By default only regular files are 43shown; use the -a option to show all file types. 44 45This script works by tracing the vfs_read() and vfs_write() functions using 46kernel dynamic tracing, which instruments explicit read and write calls. If 47files are read or written using another means (eg, via mmap()), then they 48will not be visible using this tool. 49 50This should be useful for file system workload characterization when analyzing 51the performance of applications. 52 53Note that tracing VFS level reads and writes can be a frequent activity, and 54this tool can begin to cost measurable overhead at high I/O rates. 55 56 57A -C option will stop clearing the screen, and -r with a number will restrict 58the output to that many rows (20 by default). For example, not clearing 59the screen and showing the top 5 only: 60 61# ./filetop -Cr 5 62Tracing... Output every 1 secs. Hit Ctrl-C to end 63 6408:05:11 loadavg: 0.75 0.35 0.25 3/285 822 65 66PID COMM READS WRITES R_Kb W_Kb T FILE 6732672 cksum 5006 0 320384 0 R data1 68809 run 2 0 8 0 R nsswitch.conf 69811 run 2 0 8 0 R nsswitch.conf 70804 chown 2 0 8 0 R nsswitch.conf 71 7208:05:12 loadavg: 0.75 0.35 0.25 3/285 845 73 74PID COMM READS WRITES R_Kb W_Kb T FILE 7532672 cksum 4986 0 319104 0 R data1 76845 chown 2 0 8 0 R nsswitch.conf 77828 run 2 0 8 0 R nsswitch.conf 78835 run 2 0 8 0 R nsswitch.conf 79830 run 2 0 8 0 R nsswitch.conf 80 8108:05:13 loadavg: 0.75 0.35 0.25 3/285 868 82 83PID COMM READS WRITES R_Kb W_Kb T FILE 8432672 cksum 4985 0 319040 0 R data1 85857 run 2 0 8 0 R nsswitch.conf 86858 run 2 0 8 0 R nsswitch.conf 87859 run 2 0 8 0 R nsswitch.conf 88848 run 2 0 8 0 R nsswitch.conf 89[...] 90 91This output shows a cksum command reading data1. 92 93 94An optional interval and optional count can also be added to the end of the 95command line. For example, for 1 second interval, and 3 summaries in total: 96 97# ./filetop -Cr 5 -a 1 3 98Tracing... Output every 1 secs. Hit Ctrl-C to end 99 10008:08:20 loadavg: 0.30 0.42 0.31 3/282 5187 101 102PID COMM READS WRITES R_Kb W_Kb T FILE 10312421 sshd 14101 0 225616 0 O ptmx 10412296 sshd 4 0 64 0 O ptmx 10512421 sshd 3 14104 48 778 S TCP 1065178 run 2 0 8 0 R nsswitch.conf 1075165 run 2 0 8 0 R nsswitch.conf 108 10908:08:21 loadavg: 0.30 0.42 0.31 5/282 5210 110 111PID COMM READS WRITES R_Kb W_Kb T FILE 11212421 sshd 9159 0 146544 0 O ptmx 11312421 sshd 3 9161 48 534 S TCP 11412296 sshd 1 0 16 0 S TCP 1155188 run 2 0 8 0 R nsswitch.conf 1165203 run 2 0 8 0 R nsswitch.conf 117 11808:08:22 loadavg: 0.30 0.42 0.31 2/282 5233 119 120PID COMM READS WRITES R_Kb W_Kb T FILE 12112421 sshd 26166 0 418656 0 O ptmx 12212421 sshd 4 26171 64 1385 S TCP 12312296 sshd 1 0 16 0 O ptmx 1245214 run 2 0 8 0 R nsswitch.conf 1255227 run 2 0 8 0 R nsswitch.conf 126Detaching... 127 128This example shows the -a option to include all file types. It caught heavy 129socket I/O from an sshd process, showing up as non-regular file types (the "O" 130for other, and "S" for socket, in the type column: "T"). 131 132 133USAGE message: 134 135# ./filetop -h 136usage: filetop.py [-h] [-a] [-C] [-r MAXROWS] [-p PID] [interval] [count] 137 138File reads and writes by process 139 140positional arguments: 141 interval output interval, in seconds 142 count number of outputs 143 144optional arguments: 145 -h, --help show this help message and exit 146 -a, --all-files include non-regular file types (sockets, FIFOs, etc) 147 -C, --noclear don't clear the screen 148 -r MAXROWS, --maxrows MAXROWS 149 maximum rows to print, default 20 150 -s {reads,writes,rbytes,wbytes}, --sort {reads,writes,rbytes,wbytes} 151 sort column, default rbytes 152 -p PID, --pid PID trace this PID only 153 154examples: 155 ./filetop # file I/O top, 1 second refresh 156 ./filetop -C # don't clear the screen 157 ./filetop -p 181 # PID 181 only 158 ./filetop 5 # 5 second summaries 159 ./filetop 5 10 # 5 second summaries, 10 times only 160