1Automatic regression tests:
2---------------------------
3The Valgrind gdbserver automatic tests are by default
4executed as part of the Valgrind test suite:
5   make regtest
6
7By default, these tests are running with the gdb found at
8Valgrind configure time.
9
10If you want to test with another gdb version, you can do:
11  make regtest GDB=/path/to/another/gdb
12
13or (to just run the gdbserver tests with another gdb):
14  cd gdbserver_tests
15  make check
16  cd ..
17  gdbserver_tests/make_local_links /path/to/another/gdb
18  perl tests/vg_regtest gdbserver_tests
19
20The minimum version to run the tests is gdb >= 6.5.
21    Previous versions do not have the 'target remote |' command.  It
22    would be possible to use an older version by using the option
23    --port of vgdb, and using the tcp/ip variant of the 'target remote'
24    GDB command.
25
26The tests have been run on various platforms using gdb versions >= 7.2
27and on some platforms gdb 7.0 and 7.1.
28Some gdb tests implies a gdb >= 7.2. (these are automatically disabled
29if testing with a lower version).
30Test behaviour with gdb < 7.0 is unknown: some might fail,
31some might block or loop for a very long time.
32
33Some tests implies to have a vgdb "ptrace invoker" capable.
34
35The prerequisite are established during make regtest (using marker files).
36Each test verifies the prerequisite using the prereq: line.
37
38In case of failing tests
39------------------------
40When executed with a new gdb version and/or depending on the OS version,
41gdbserver tests might often fail due to (irrelevant) differences.
42Such irrelevant differences have to be filtered by gdbserver_tests/filter_gdb.
43
44You are welcome to fix such bugs by enhancing filter_gdb.
45
46Alternatively, to report such problems, the best is to re-run
47the gdbserver tests the following way:
48   perl tests/vg_regtest --keep-unfiltered gdbserver_tests
49
50Then file a bug in bugzilla, giving the following information:
51  output of
52    gdbserver_tests/gdb --version
53    uname -a
54    cat /etc/issue
55    valgrind --version (and/or svn version)
56  and attach a tar file containing all the *.out and *.diff
57  files in gdbserver_tests directory
58
59If a gdbserver test fails for other reasons, you can run the test
60manually in two windows:
61  In one window, the valgrind
62  In another window, you launch gdb yourself, and you copy paste
63  the command from xxxx.stdinB.gdb. This might help to see what is
64  wrong.
65
66Another good trick is also to execute the same kind of actions
67using a gdb connected to the gdbserver part of gdb.
68You can examine what is happening by enabling the trace
69of the packets being sent using the gdb command:
70   set debug remote 1
71Note however that the packets might be different
72(e.g. the Valgrind gdbserver understands the 'P' packet,
73which might not be understood by the gdbserver of gdb).
74
75Naming conventions:
76-------------------
77
78The gdbserver tests are done with various Valgrind tools.  A gdbserver
79test using a tool xxxxxx should have its name starting with the 'short
80two letters code' of this tool.  For example, the test mcvabits.vgtest
81is using Memcheck tool, while the test mssnapshot.vgtest is using
82massif tool.
83
84Typically, a gdbserver test implies to launch two programs:
85     prog: a program running under valgrind
86     progB: another program (typically, either gdb or vgdb standalone)
87The conventions about how to specify the 2nd program in a .vgtest
88are explained in the file ../tests/vg_regtest.in
89Many tests are using gdb as progB. The input for gdb is named
90xxxxxx.stdinB.gdb.
91
92One day, we might imagine to run tests in parallel.
93So, each test gets its own '--vgdb-prefix' argument.
94This also help to avoid interactions between two successive tests:
95if a previous test stayed blocked, the vgdb of the next test
96will not report an error due to multiple possible FIFOs.
97
98
99Rational for test approach
100--------------------------
101Two approaches have been looked at:
102  V: use the 'vg_regtest' approach used by the rest of Valgrind tests
103  G: use the gdb Dejagnu test framework.
104
105Advantages of V: much simpler that G, known by Valgrind developers,
106no additional dependency for the Valgrind build and test.
107
108Disadvantages of V: not well suited to testing of interactive tools,
109very unflexible way to test the results (everything is in "template"
110files), templates contains often irrelevant data for the test but it
111can make the test fail.  After writing 13 tests, it looks like the
112template diff approach is quite fragile (e.g. changing the gdb version
113and/or the OS version influences the output of irrelevant things which
114are part of the template).
115
116A more flexible template filtering is needed.
117Maybe something like:
118The program under test is outputting its instructions to be filtered in
119special markers e.g.
120#pushf filter_addresses | filter_messages
121... some output
122#pushf an_additional_filter
123... some other output, filtered by both the first and second push
124#popf
125... here output filtered only by the first pushf
126#popf
127
128Advantages of G: much more powerful, well suited to test a gdb with a
129gdbserver, tests can verify specifically some output without being
130impacted by other output, allow to test Valgrind gdbserver with the
131all of the gdb test suite (but a lot of tests will rather test gdb
132than Valgrind gdbserver).
133
134Disadvantages: not an easy beast to understand and master, running the
135whole gdb testsuite with Valgrind gdbserver looks to be a challenge.
136
137Currently, tests are written with vg_regtest. Approach G will be looked at it
138again (e.g. to complement V) once a basic set of tests are available.
139
140
141Manual tests still to automate:
142-------------------------------
143
144Validate  monitor commands abbreviation recognition
145***************************************************
146mo v.info all_errors            # to show all errors recorded so far
147mo v.i a                        # the same
148mo v                            # must give an error: v can match v.set v.info
149mo v                            # the same            v
150mo v.                           # the same            v.
151
152test of gdb detaching or dying
153******************************
154valgrind --vgdb=yes --vgdb-error=0 --vgdb-poll=500 ./t
155
156in another window
157
158gdb ./t
159set remotetimeout 100
160target remote|vgdb
161detach                         valgrind continues
162target remote|vgdb             reattach
163detach                         valgrind continues
164target remote|vgdb             reattach
165monitor v.wait                 no effect
166
167
168
169test of valgrind/gdb output redirection
170***************************************
171valgrind --vgdb=yes --vgdb-error=1 --vgdb-poll=500 ./t
172
173in another window
174
175**** command to type***         ****** expected behaviour
176gdb ./t
177set remotetimeout 1000
178target remote | vgdb
179mo v.set vgdb-error 1000        # so that valgrind does stop only at error 1000 and after
180mo v.set gdb_output             # to have further valgrind errors output in gdb
181c                               # continue, some errors will appear
182C-c                             # interrupt program
183mo v.set log_output             # to set back the valgrind output to normal process log output
184mo l                            # leak output to appear in log of process
185mo v.set mixed_output
186mo l                            # leak output to appear in gdb
187
188
189
190test with a big executable: firefox
191***********************************
192valgrind --vgdb=yes --vgdb-error=1000 --vgdb-poll=50000 --trace-children=yes firefox 2>&1 | tee f.out
193
194wait for some messages from the "big" firefox executable to appear.
195Then:
196
197gdb /usr/lib/firefox-3.5/firefox
198target remote | vgdb
199... then you can do various next/print/bt/bt full/break/... to see it is working
200
201bulk test with the above
202************************
203to verify there is no race condition/no reentrance problem
204between gdbserver code and valgrind:
205start firefox like in the previous test.
206In another window, do:
207  while true
208  do
209    vgdb leak
210    sleep 1
211  done
212
213NB: this will make firefox run extremely slow, as it will do a leak
214search every second.
215
216
217Test of "jump" functionality
218----------------------------
219... to be done : put two breaks, jump over one.
220... same but when error is encountered
221
222
223* test with --max-invoke-ms=0
224-----------------------------
225valgrind --vgdb=yes ./t
226... wait till you see "petachounok sleeping 4 of 15
227then try to gdb it
228
229!!!! this often causes gdb to report a protocol timeout.
230use gdb set remotetimeout <a big time> to avoid that.
231The symptoms of a timeout are:
232    (gdb) tar rem|vgdb --max-invoke-ms=0
233    Remote debugging using |vgdb --max-invoke-ms=0
234    relaying data between gdb and process 2930
235    Ignoring packet error, continuing...
236    warning: unrecognized item "timeout" in "qSupported" response
237
238* tests of shadow registers
239----------------------------
240Show/modify shadow registers
241