1Demonstrations of slabratetop, the Linux eBPF/bcc version.
2
3
4slabratetop shows the rate of allocations and total bytes from the kernel
5memory allocation caches (SLAB or SLUB), in a top-like display that refreshes.
6For example:
7
8# ./slabratetop
9<screen clears>
1007:01:35 loadavg: 0.38 0.21 0.12 1/342 13297
11
12CACHE                            ALLOCS      BYTES
13kmalloc-4096                       3554   14557184
14kmalloc-256                        2382     609536
15cred_jar                           2568     493056
16anon_vma_chain                     2007     128448
17anon_vma                            972      77760
18sighand_cache                        24      50688
19mm_struct                            49      50176
20RAW                                  52      49920
21proc_inode_cache                     59      38232
22signal_cache                         24      26112
23dentry                              135      25920
24sock_inode_cache                     29      18560
25files_cache                          24      16896
26inode_cache                          13       7696
27TCP                                   2       3840
28pid                                  24       3072
29sigqueue                             17       2720
30ext4_inode_cache                      2       2160
31buffer_head                          16       1664
32xfs_trans                             5       1160
33
34By default the screen refreshes every one second, and only the top 20 caches
35are shown. These can be tuned with options: see USAGE (-h).
36
37The output above showed that the kmalloc-4096 cache allocated the most, about
3814 Mbytes during this interval. This is a generic cache; other caches have
39more meaningful names ("dentry", "TCP", "pid", etc).
40
41slabtop(1) is a similar tool that shows the current static volume and usage
42of the caches. slabratetop shows the active call rates and total size of the
43allocations.
44
45
46Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the
47kernel stacks that led to this allocation. In the future (maybe by now) the
48bcc trace tool could do this. As I'm writing this, it can't, so I'll use my
49older ftrace-based kprobe tool as a workarond. This is from my perf-tools
50collection: https://github.com/brendangregg/perf-tools.
51
52# ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100
53Tracing kprobe kmem_cache_alloc. Ctrl-C to end.
54          kprobe-3892  [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
55          kprobe-3892  [002] d... 7888274.478333: <stack trace>
56 => kmem_cache_alloc
57 => user_path_at_empty
58 => vfs_fstatat
59 => SYSC_newstat
60 => SyS_newstat
61 => entry_SYSCALL_64_fastpath
62          kprobe-3892  [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
63          kprobe-3892  [002] d... 7888274.478341: <stack trace>
64 => kmem_cache_alloc
65 => user_path_at_empty
66 => vfs_fstatat
67 => SYSC_newstat
68 => SyS_newstat
69 => entry_SYSCALL_64_fastpath
70          kprobe-3892  [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
71          kprobe-3892  [002] d... 7888274.478346: <stack trace>
72 => kmem_cache_alloc
73 => user_path_at_empty
74 => vfs_fstatat
75 => SYSC_newstat
76 => SyS_newstat
77 => entry_SYSCALL_64_fastpath
78          kprobe-3892  [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
79          kprobe-3892  [002] d... 7888274.478351: <stack trace>
80 => kmem_cache_alloc
81 => user_path_at_empty
82 => vfs_fstatat
83 => SYSC_newstat
84 => SyS_newstat
85 => entry_SYSCALL_64_fastpath
86          kprobe-3892  [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
87          kprobe-3892  [002] d... 7888274.478355: <stack trace>
88 => kmem_cache_alloc
89 => user_path_at_empty
90 => vfs_fstatat
91 => SYSC_newstat
92 => SyS_newstat
93 => entry_SYSCALL_64_fastpath
94          kprobe-3892  [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
95          kprobe-3892  [002] d... 7888274.478359: <stack trace>
96 => kmem_cache_alloc
97 => user_path_at_empty
98 => vfs_fstatat
99 => SYSC_newstat
100 => SyS_newstat
101 => entry_SYSCALL_64_fastpath
102[...]
103
104This is just an example so that you can see it's possible to dig further.
105Please don't copy-n-paste that kprobe command, as it's unlikely to work (the
106"+0(+96(%di))" text is specific to a kernel version and architecture).
107
108So these allocations are coming from user_path_at_empty(), which calls other
109functions (not seen in the stack: I suspect it's a tail-call compiler
110optimization).
111
112
113USAGE:
114
115# ./slabratetop -h
116usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count]
117
118Kernel SLAB/SLUB memory cache allocation rate top
119
120positional arguments:
121  interval              output interval, in seconds
122  count                 number of outputs
123
124optional arguments:
125  -h, --help            show this help message and exit
126  -C, --noclear         don't clear the screen
127  -r MAXROWS, --maxrows MAXROWS
128                        maximum rows to print, default 20
129
130examples:
131    ./slabratetop            # kmem_cache_alloc() top, 1 second refresh
132    ./slabratetop -C         # don't clear the screen
133    ./slabratetop 5          # 5 second summaries
134    ./slabratetop 5 10       # 5 second summaries, 10 times only
135