1Tests naming convention
2-----------------------
3Names of tests (suffixes of control.* files) are build from 2 or 3 words
4separated by an underscore character. The first one is '100' or 'all', what
5defines the set of PPD files to test: '100' means 100 chosen (most often used)
6PPD files (stored in the archive ppds_100) while 'all' means all available PPD
7files (stored in the archive ppds_all). The second word of the name is either
8'test' or 'dump'. The third and optional word is 'debug'. All these three words
9influence testing procedure used by the autotest, their meanings are described
10in next paragraphs.
11
12Overall testing procedure
13-------------------------
14The purpose of this autotest is to verify that given subset of PPD files work
15in ChromeOS. Each PPD file is tested with the following procedure:
161. A printer driver is added to CUPS server.
172. Test documents are printed on the configured printer.
183. Raw output from the CUPS server is intercepted by, so called, FakePrinter.
194. CUPS logs are parsed to make sure that no errors occured.
205. Obtained outputs are verified (see below) - test only.
216. A printing pipeline used by CUPS is rerun in shell (see below) - debug only.
227. All obtained outputs & logs are saved on the device (see below) - dump only.
238. The printer driver is removed from CUPS server.
24This procedure is repeated for every PPD file. The number of PPD files may be
25large (3K+ files). To decrease amount of time needed by the autotest, several
26PPD files are tested simultaneously in parallel threads. Autotest always run
27the procedure for all given PPD files and print a summary report at the end.
28If at least one of PPD files fails, whole autotest is finished with failure
29(but always all PPD files are processed).
30
31Output verification ('test')
32----------------------------
33Intercepted output is verified by comparision with the previous results
34obtained for the same PPD. We cannot store outputs directly, because their
35total size may have hundreds of megabytes. Instead of that short digest is
36calculated for each obtained document and it is used for comparison.
37A function for digests calculation is in the 'helpers.py' file. Not all
38outputs can be tested this way because for some PPD files produced contents
39differ between runs. List of PPD files for which we cannot calculate
40constant digest is saved in the file digests_blacklist.txt. Files with
41expected digests for every test document are stored in the directory "digests".
42If a digests for given pair (test document, PPD file) is missing, the test
43checks only check if the output is not empty (or not too short).
44
45Rerun printing pipeline ('debug')
46---------------------------------
47Every PPD file defines printing pipeline used by CUPS, consisting of external
48binaries (so called filters). To make sure that every part of the printing
49pipeline works correctly, it can be run outside CUPS in Linux shell. In tests
50with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In
51this mode every document is processed twice: the first time by CUPS and the
52second time by extracted pipeline run as bash script. Additionally, each
53pipeline's step is verified by checking its return code (it is supposed to be
54equal 0) and the final output is compared with the output returned by CUPS
55server. If the test is also defined as 'dump', each intermediate content is
56saved to the directory given in path_outputs parameter.
57This mode requires more disk space on tested device (~2GB in /tmp) and takes
58more execution time (~2 times more). All information needed to reconstruct
59a pipeline used by CUPS can are extracted from CUPS logs after the job is
60completed. To do that, CUPS configuration must be modified: the LogLevel
61option in /etc/cups/cupsd.conf must be set to "debug". This modification is
62done when the autotest is initialized; the original setting is restored in
63the cleaup stage.
64
65
66Save outputs and logs ('dump')
67------------------------------
68All obtained outputs and logs are saved on the device in the directory
69/tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix
70are grouped together and stored in single archive to limit usage of disk space.
71
72Test parameters
73---------------
74path_docs - path to directory with test documents (PDF files)
75path_ppds - path to directory with PPD files, if not set then all available PPD
76            files are downloaded and tested
77path_digests - path to directory with files containing digests for
78            verification, if not set then outputs are not verified
79path_outputs - if set, then all outputs are dumped there (given directory is
80            deleted if already exists); also all digests files are recalculated
81            and saved in the same directory
82debug_mode - True or False, if True then for every test the printing pipeline
83            is extracted from CUPS logs and executed again outside CUPS
84
85Generating new digests
86----------------------
87The following procedure can be used to update digests:
881. Run the test defined in control.all_dump:
89        test_that <device IP>  platform_PrinterPpds.all_dump
902. Download generated files with digests to your workstation
91        rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir>
923. Replace the files from the "digests" directory and commit changes
93
94Updating the archive with PPD files
95-----------------------------------
96Currently, all tests are based on PPD files stored in local directories. The
97autotest can download all PPD files by itself, but we do not use this option
98to limit the number of possible points of failures during testing. PPD files
99are stored in the archive 'ppds_all.tar.xz'. To replace the archive with the
100current list of supported PPD files, one can use the script
101'download_ppds_make_archive.py'.
102
103Others
104------
105* How to get comma-separated list of all used ghostscript devices?
106  1. Go to the directory generated by all_dump_debug
107  2. Unpack all tar.xz archives (not on the device)
108        for d in *.pdf;
109        do
110            cd $d;
111            for f in *.tar.xz; do tar xf $f; done ;
112            cd ..;
113        done
114  3. Run the following
115        grep -o sDEVICE=[^\ ]*\  ./*.pdf/*.err[12345] | cut -d \= -f 2 | sort | uniq | tr -d ' ' | tr '\n' ','
116  4. Add to the list uniprint (it is not caught by the command above)
117* How to get a list of all used cups filters?
118  1. Go to the directory generated by all_dump_debug
119  2. Unpack all tar.xz archives (not on the device)
120        for d in *.pdf;
121        do
122            cd $d;
123            for f in *.tar.xz; do tar xf $f; done ;
124            cd ..;
125        done
126  3. Run the following
127        grep -o '(exec -a "Fake[^"]*" [^ ]* ' ./*.pdf/*.sh | cut -d \  -f 4 | sort | uniq
128