Merge branch 'dbmjz'
[exim.git] / test / README
CommitLineData
151b83f8
PH
1EXPORTABLE EXIM TEST SUITE
2--------------------------
3
4This document last updated for:
5
bbe15da8 6Test Suite Version: 4.67
a14e5636 7Date: 20 February 2007
151b83f8
PH
8
9
10BACKGROUND
11----------
12
13For a long time, the Exim test suite was confined to Philip Hazel's
14workstation, because it relied on that particular environment. The problem is
15that an MTA such as Exim interacts a great deal with its environment, so if you
16run it somewhere else, the output will be different, which makes automatic
17checking difficult. Even in a single environment, things are not all that easy.
18For instance, if Exim delivers a message, the log line (which one would want to
19compare) contains a timestamp and an Exim message id that will be different
20each time. This issue is dealt with by a Perl script that munges the output by
21recognizing changing sequences and replacing them with fixed values before
22doing a comparison. Another problem with exporting the original test suite is
23that it assumes a version of Exim with more or less every optional feature
24enabled.
25
26This README describes a new test suite that is intended to be exportable and to
1b781f48
PH
27run in a number of different environments. The tests themselves are in no
28particular order; they accumulated over the years as Exim was extended and
29modified. They vary greatly in size and complexity. Some were specifically
30constructed to test new features; others were made to demonstrate that a bug
31had been fixed.
151b83f8
PH
32
33A few of the original tests have had to be omitted from this more general
34suite because differences in operating system behaviour make it impossible to
35generalize them. An example is a test that uses a version of Exim that is
36setuid to the Exim user rather than root, with the deliver_drop_privilege
37option set. In Linux, such a binary is able to deliver a message as the caller
38of Exim, because it can revert to the caller's uid. In FreeBSD this is not the
39case.
40
151b83f8
PH
41
42REQUIREMENTS
43------------
44
45In order to run this test suite, the following requirements must be met:
46
1b781f48 47(1) You should run the tests on a matching version of Exim, because the suite
151b83f8
PH
48 is continuously updated to test the latest features and bug fixes. The
49 version you test does not, however, have to be installed as the live
1b781f48
PH
50 version. You can of course try the tests on any version of Exim, but some
51 may fail. In particular, the test suite will fall apart horrible with
52 versions of Exim prior to 4.54.
151b83f8
PH
53
54(2) You can use any non-root login to run the tests, but there must be access
55 via "sudo" to root from this login. Privilege is required to override
56 configuration change checks and for things like cleaning up spool files,
57 but on the other hand, the tests themselves need to call Exim from a
58 non-root process. The use of "sudo" is the easiest way to achieve all this.
59 The test script uses "sudo" to do a number of things as root, so it is best
60 if you set a sudo timeout so that you do not have to keep typing a
61 password. For example, if you put
62
63 Defaults timestamp_timeout=480
64
65 in /etc/sudoers, a password lasts for 8 hours (a working day). It is
a56f166d
JJ
66 not permitted to run the tests as the Exim user because the test suite
67 tracks the two users independently. Using the same user would result
68 in false positives on some tests.
151b83f8 69
8121f028
PP
70 Further, some tests invoke sudo in an environment where there might not be
71 a TTY, so tickets should be global, not per-TTY. Taking this all together
72 and assuming a user of "exim-build", you might have this in sudoers:
73
74 Defaults:exim-build timestamp_timeout=480,!tty_tickets
75
151b83f8
PH
76(3) The login under which you run the tests must be in the exim group so that
77 it has access to logs, spool files, etc. The login should not be one of the
78 names "userx", "usery", "userz", or a few other simple ones such as "abcd"
1b781f48
PH
79 and "xyz" and single letters that are used in the tests. The test suite
80 expects the login to have a gecos name; I think it will now run if the
81 gecos field is empty but there may be anomalies.
151b83f8
PH
82
83(4) The directory into which you unpack the test suite must be accessible by
1b781f48
PH
84 the Exim user, so that code running as exim can access the files therein. A
85 world-readable directory is fine. However, there may be problems if the
86 path name of the directory is excessively long. This is because it
87 sometimes appears in log lines or debug output, and if it is truncated, it
151b83f8
PH
88 is no longer recognized.
89
90(5) Exim must be built with its user and group specified at build time, and
91 with certain minimum facilities, namely:
92
93 Routers: accept, dnslookup, manualroute, redirect
94 Transports: appendfile, autoreply, pipe, smtp
95 Lookups: lsearch
96
97 Most Exim binaries will have these included.
98
99(6) A C compiler is needed to build some test programs, and the test script is
100 written in Perl, so you need that.
101
102(7) Some of the tests run Exim as a daemon, and others use a testing server
103 (described below). These require TCP ports. In the configurations and
104 scripts, the ports are parameterized, but at present, fixed values are
105 written into the controlling script. These are ports 1224 to 1229. If these
106 ports are not available for use, some of the tests will fail.
107
108(8) There is an underlying assumption that the host on which the tests are
109 being run has an IPv4 address (which the test script seeks out). If there
110 is also an IPv6 address, additional tests are run when the Exim binary
111 contains IPv6 support. There are checks in the scripts for a running IPv4
112 interface; when one is not found, some tests are skipped (with a warning
113 message).
114
33191679
PP
115(9) Exim must be built with TRUSTED_CONFIG_LIST support, so that the test
116 configs can be placed into it. DISABLE_D_OPTION must not be used. If
117 ALT_CONFIG_PREFIX is used, it must contain the directory of the test-suite.
118 WHITELIST_D_MACROS should contain:
119
120 DIR:EXIM_PATH:AA:ACL:ACLRCPT:ACL_MAIL:ACL_PREDATA:ACL_RCPT:AFFIX:ALLOW:ARG1:ARG2:AUTHF:AUTHS:AUTH_ID_DOMAIN:BAD:BANNER:BB:BR:BRB:CERT:COM:COMMAND_USER:CONNECTCOND:CONTROL:CREQCIP:CREQMAC:CRL:CSS:D6:DATA:DCF:DDF:DEFAULTDWC:DELAY:DETAILS:DRATELIMIT:DYNAMIC_OPTION:ELI:ERROR_DETAILS:ERT:FAKE:FALLBACK:FILTER:FILTER_PREPEND_HOME:FORBID:FORBID_SMTP_CODE:FUSER:HAI:HAP:HARDLIMIT:HEADER_LINE_MAXSIZE:HEADER_MAXSIZE:HELO_MSG:HL:HOSTS:HOSTS_AVOID_TLS:HOSTS_MAX_TRY:HVH:IFACE:IGNORE_QUOTA:INC:INSERT:IP1:IP2:LAST:LDAPSERVERS:LENCHECK:LIMIT:LIST:LOG_SELECTOR:LS:MAXNM:MESSAGE_LOGS:MSIZE:NOTDAEMON:ONCE:ONLY:OPT:OPTION:ORDER:PAH:PEX:PORT:PTBC:QDG:QOLL:QUOTA:QUOTA_FILECOUNT:QWM:RCPT_MSG:REMEMBER:REQUIRE:RETRY:RETRY1:RETRY2:RETURN:RETURN_ERROR_DETAILS:REWRITE:ROUTE_DATA:RRATELIMIT:RT:S:SELECTOR:SELF:SERVER:SERVERS:SREQCIP:SREQMAC:SRV:STD:STRICT:SUB:SUBMISSION_OPTIONS:TIMEOUTDEFER:TIMES:TRUSTED:TRYCLEAR:UL:USE_SENDER:UTF8:VALUE:WMF:X:Y
121
4b9529fc
PP
122(10) Exim must *not* be built with USE_READLINE, as the test-suite's automation
123 assumes the simpler I/O model.
daea6332 124 Exim must *not* be built with HEADERS_CHARSET set to UTF-8.
4b9529fc 125
33191679 126
151b83f8
PH
127
128OPTIONAL EXTRAS
129---------------
130
131If the Exim binary that is being tested contains extra functionality in
132addition to the minimum specified above, additional tests are run to exercise
133the extra functionality, except for a few special cases such as the databases
134(MySQL, PostgreSQL, LDAP) where special data is needed for the tests.
135
136
137RUNNING THE TEST SUITE
138----------------------
139
140(1) Download the tarball exim-testsuite-x.xx.tar.bz2 and unpack it, preferably
141 in a directory alongside an Exim source directory (see below).
142
143(2) cd into the exim-testsuite-x.xx directory.
144
145(3) Run "./configure" and then "make". This builds a few auxiliary programs
146 that are written in C.
147
8121f028 148(4) echo $PWD/test-config >> your_TRUSTED_CONFIG_LIST_filename
33191679
PP
149
150(5) Run "./runtest" (a Perl script) as described below.
151b83f8 151
33191679 152(6) If you want to see what tests are available, run "./listtests".
151b83f8
PH
153
154
155BREAKING OUT OF THE TEST SCRIPT
156-------------------------------
157
158If you abandon the test run by typing ^C, the interrupt may be passed to a
159program that the script is running, or it may be passed to the script itself.
160In the former case, the script should detect that the program has ended
161abnormally. In both cases, the script tries to clean up everything, including
162killing any Exim daemons that it has started. However, there may be race
163conditions in which the clean up does not happen. If, after breaking out of a
164run, you see strange errors in the next run, look for any left-over Exim
165daemons, and kill them by hand.
166
167
168THE LISTTESTS SCRIPT
169--------------------
170
171The individual test scripts are in subdirectories of the "scripts" directory.
172If you do not supply any arguments to ./listtests, it scans all the scripts in
173all the directories, and outputs the heading line from each script. The output
174is piped through "less", and begins like this:
175
176=== 0000-Basic ===
177Basic/0001 Basic configuration setting
178Basic/0002 Common string expansions
179Basic/0003 Caseless address blocking
180...
181
182Lines that start === give the name of the subdirectory containing the test
183scripts that follow. If you supply an argument to ./listtests, it is used as a
184Perl pattern to match case-independently against the names of the
185subdirectories. Only those that match are scanned. For example, "./listtests
186ipv6" outputs this:
187
188=== 1000-Basic-ipv6 ===
189=== Requires: support IPv6
190Basic-ipv6/1000 -bh and non-canonical IPv6 addresses
191Basic-ipv6/1001 recognizing IPv6 address in HELO/EHLO
192
193=== 2250-dnsdb-ipv6 ===
194=== Requires: support IPv6
195 lookup dnsdb
196dnsdb-ipv6/2250 dnsdb ipv6 lookup in string expansions
197
198If you supply a second argument to ./listtests, it is used as a Perl pattern to
199match case-independently against the individual script titles. For example,
200"./listtests . mx" lists all tests whose titles contain "mx", because "."
201matches all the subdirectory names.
202
203
204THE RUNTEST SCRIPT
205------------------
206
207If you do not supply any arguments to ./runtest, it searches for an Exim
208source tree at the same level as the test suite directory. It then looks for an
209Exim binary in a "build" directory of that source tree. If there are several
210Exim source trees, it chooses the latest version of Exim. Consider the
211following example:
212
213 $ ls -F /source/exim
1b781f48 214 exim-4.60/ exim-4.62/ exim-testsuite-x.xx/
151b83f8 215
1b781f48
PH
216A simple ./runtest from within the test suite will use a 4.62 binary if it
217finds one, otherwise a 4.60 binary. If a binary cannot be found, the script
151b83f8
PH
218prompts for one. Alternatively, you can supply the binary on the command line:
219
220 ./runtest /usr/exim/bin/exim
221
1b781f48
PH
222A matching test suite is released with each Exim release; if you use a test
223suite that does not match the binary, some tests may fail.
224
225The test suite uses some of the Exim utilities (such as exim_dbmbuild), and it
226expects to find them in the same directory as Exim itself. If they are not
227found, the tests that use them are omitted. A suitable comment is output.
151b83f8
PH
228
229On the ./runtest command line, following the name of the binary, if present,
230there may be a number of options and then one or two numbers. The full syntax
231is as follows:
232
233 ./runtest [binary name] [runtest options] [exim options] \
234 [first test] [last test]
235
236There are some options for the ./runtest script itself:
237
238 -DEBUG This option is for debugging the test script. It causes some
239 tracing information to be output.
240
241 -DIFF By default, file comparisons are done using a private compare
242 command called "cf", which is built from source that is provided in
243 the src directory. This is a command I've had for nearly 20 years -
244 look at the source comments for its history - whose output I
245 prefer. However, if you want to use "diff" instead, give -DIFF as a
246 runtest option. In that case, "diff -u" is used for comparisons.
247 (If it turns out that most people prefer to use diff, I'll change
248 the default.)
249
250 -KEEP Normally, after a successful run, the test output files are
251 deleted. This option prevents this. It is useful when running a
252 single test, in order to look at the actual output before it is
253 modified for comparison with saved output.
254
255 -NOIPV4 Pretend that an IPv4 interface was not found. This is useful for
256 testing that the test suite correctly skips tests that require
257 a running IPv4 interface.
258
259 -NOIPV6 Pretend that an IPv6 interface was not found. This is useful for
260 testing that the test suite correctly skips tests that require
261 a running IPv6 interface.
262
263 -UPDATE If this option is set, any detected changes in test output are
264 automatically accepted and used to update the stored copies of the
265 output. It is a dangerous option, but it useful for the test suite
266 maintainer after making a change to the code that affects a lot of
267 tests (for example, the wording of a message).
268
269The options for ./runtest must be given first (but after the name of the
270binary, if present). Any further options, that is, items on the command line
271that start with a hyphen, are passed to the Exim binary when it is run as part
272of a test. The only sensible use of this is to pass "-d" in order to run a test
273with debugging enabled. Any other options are likely to conflict with options
274that are set in the tests. Some tests are already set up to run with debugging.
275In these cases, -d on the command line overrides their own debug settings.
276
277The final two arguments specify the range of tests to be run. Test numbers lie
278in the range 1 to 9999. If no numbers are given, the defaults are 1 and 8999
279(sic). Tests with higher numbers (9000 upwards) are not run automatically
280because they require specific data (such as a particular MySQL table) that is
281unlikely to be generally available.
282
283Tests that require certain optional features of Exim are grouped by number, so
284in any given range, not all the tests will exist. Non-existent tests are just
285skipped, but if there are no tests at all in the given range, a message is
286output.
287
288If you give only one number, just that test is run (if it exists). Instead of a
289second number, you can give the character "+", which is interpreted as "to the
290end". Normally this is 8999; if the starting number is 9000 or higher, "+" is
291interpreted as 9999. Examples:
292
293 ./runtest 1300
294 ./runtest 1400 1699
295 ./runtest /usr/sbin/exim 5000 +
296 ./runtest -DIFF -d 81
297
298When the script starts up, the first thing it does is to check that you have
299sudo access to root. Then it outputs the version number of the Exim binary that
300it is testing, and also information about the optional facilities that are
301present (obtained from "exim -bV"). This is followed by some environmental
302information, including the current login id and the hosts's IP address. The
303script checks that the current user is in the Exim group, and that the Exim
304user has access to the test suite directory.
305
306The script outputs the list of tests requested, and a list of tests that will
307be omitted because the relevant optional facilities are not in the binary. You
308are then invited to press Return to start the tests running.
309
310
311TEST OUTPUT
312-----------
313
314When all goes well, the only permanent output is the identity of the tests as
315they are run, and "Script completed" for each test script, for example:
316
317 Basic/0001 Basic configuration setting
318 Script completed
319 Basic/0002 Basic string expansions
320 Script completed
321 Basic/0003 Caseless address blocking
322 Script completed
323 Basic/0004 Caseful address blocking
324 Script completed
325 Basic/0005 -bs to simple local delivery
326 ...
327
328While a script is running, it shows "Test n" on the screen, for each of the
329Exim tests within the script. There may also be comments from some tests when a
330delay is expected, for example, if there is a "sleep" while testing a timeout.
331
332Before each set of optional tests, an extra identifying line is output. For
333example:
334
335 >>> The following tests require: authenticator cram_md5
336 CRAM-MD5/2500 CRAM-MD5 server tests
337 Script completed
338 CRAM-MD5/2501 CRAM-MD5 client tests
339 Script completed
340
341If a test fails, you are shown the output of the text comparison that failed,
342and prompted as to what to do next. The output is shown using the "less"
1b781f48
PH
343command, or "more" if "less" is not available. The options for "less" are set
344to that it automatically exits if there is less that a screenful of output. By
345default, the output is from the "cf" program, and might look like this:
151b83f8
PH
346
347 DBM/1300 DBM files and exim_dbmbuild
348 ===============
349 Lines 7-9 of "test-stdout-munged" do not match lines 7-11 of "stdout/1300".
350 ----------
351 exim_dbmbuild exit code = 1
352 Continued set of lines is too long: max permitted length is 99999
353 exim_dbmbuild exit code = 1
354 ----------
355 dbmbuild abandoned
356 exim_dbmbuild exit code = 2
357 Continued set of lines is too long: max permitted length is 99999
358 dbmbuild abandoned
359 exim_dbmbuild exit code = 2
360 ===============
361 1 difference found.
362 "test-stdout-munged" contains 16 lines; "stdout/1300" contains 18 lines.
363
364 Continue, Update & retry, Quit? [Q]
365
366This example was generated by running the test with a version of Exim
367that had a bug in the exim_dbmbuild utility (the bug was fixed at release
3684.53). See "How the tests work" below for a description of the files that are
369used. In this case, the standard output differed from what was expected.
370
371The reply to the prompt must either be empty, in which case it takes the
372default that is given in brackets (in this case Q), or a single letter, in
373upper or lower case (in this case, one of C, U, or Q). If you type anything
374else, the prompt is repeated.
375
376"Continue" carries on as if the files had matched; that is, it ignores the
377mismatch. Any other output files for the same test will be compared before
378moving on to the next test.
379
380"Update & retry" copies the new file to the saved file, and reruns the test
381after doing any further comparisons that may be necessary.
382
383Other circumstances give rise to other prompts. If a test generates output for
384which there is no saved data, the prompt (after a message stating which file is
385unexpectely not empty) is:
386
387 Continue, Show, or Quit? [Q]
388
389"Show" displays the data on the screen, and then you get the "Continue..."
390prompt. If a test ends with an unexpected return code, the prompt is:
391
392 show stdErr, show stdOut, Continue (without file comparison), or Quit? [Q]
393
394Typically in these cases there will be something interesting in the stderr
395or stdout output. There is a similar prompt after the "server" auxiliary
396program fails.
397
398
399OPENSSL AND GNUTLS ERROR MESSAGES
400---------------------------------
401
402Some of the TLS tests deliberately cause errors to check how Exim handles them.
403It has been observed that different releases of the OpenSSL and GnuTLS
404libraries generate different error messages. This may cause the comparison with
405the saved output to fail. Such errors can be ignored.
406
407
1b781f48
PH
408OTHER ISSUES
409------------
410
411. Some of the tests are time-sensitive (e.g. when testing timeouts, as in test
412 461). These may fail if run on a host that is also running a lot of other
413 processes.
414
415. Some versions of "ls" use a different format for times and dates. This can
416 cause test 345 to fail.
417
418. Test 0142 tests open file descriptors; on some hosts the output may vary.
419
420
151b83f8
PH
421OTHER SCRIPTS AND PROGRAMS
422--------------------------
423
424There is a freestanding Perl script called "listtests" that scans the test
425scripts and outputs a list of all the tests, with a short descriptive comment
426for each one. Special requirements for groups of tests are also noted.
427
428The main runtest script makes use of a second Perl script and some compiled C
429programs. These are:
430
431patchexim A Perl script that makes a patched version of Exim (see the
432 next section for details).
433
434bin/cf A text comparison program (see above).
435
436bin/checkaccess A program that is run as root; it changes uid/gid to the
437 Exim user and group, and then checks that it can access
438 files in the test suite's directory.
439
440bin/client A script-driven SMTP client simulation.
441
442bin/client-gnutls A script-driven SMTP client simulation with GnuTLS support.
443 This is built only if GnuTLS support is detected on the host.
444
445bin/client-ssl A script-driven SMTP client simulation with OpenSSL support.
446 This is built only if OpenSSL support is detected on the
447 host.
448
449bin/fakens A fake "nameserver" for DNS tests (see below for details).
450
451bin/fd A program that outputs details of open file descriptors.
452
453bin/iefbr14 A program that does nothing, and returns 0. It's just like
454 the "true" command, but it is in a known place.
455
456bin/loaded Some dynamically loaded functions for testing dlfunc support.
457
bbe15da8
PH
458bin/mtpscript A script-driven SMTP/LMTP server simulation, on std{in,out}.
459
460bin/server A script-driven SMTP server simulation, over a socket.
461
462bin/showids Output the current uid, gid, euid, egid.
151b83f8
PH
463
464The runtest script also makes use of a number of ordinary commands such as
465"cp", "kill", "more", and "rm", via the system() call. In some cases these are
466run as root by means of sudo.
467
468
469STANDARD SUBSTITUTIONS
470----------------------
471
472In the following sections, there are several references to the "standard
473substitutions". These make changes to some of the stored files when they are
474used in a test. To save repetition, the substitutions themselves are documented
475here:
476
477 CALLER is replaced by the login name of the user running the tests
1b781f48 478 CALLERGROUP is replaced by the caller's group id
151b83f8
PH
479 CALLER_GID is replaced by the caller's group id
480 CALLER_UID is replaced by the caller's user id
481 DIR is replaced by the name of the test-suite directory
482 EXIMGROUP is replaced by the name of the Exim group
483 EXIMUSER is replaced by the name of the Exim user
484 HOSTIPV4 is replaced by the local host's IPv4 address
485 HOSTIPV6 is replaced by the local host's IPv6 address
486 HOSTNAME is replaced by the local host's name
487 PORT_D is replaced by a port number for normal daemon use
488 PORT_N is replaced by a port number that should never respond
489 PORT_S is replaced by a port number for normal bin/server use
490 TESTNUM is replaced by the current test number
491 V4NET is replaced by an IPv4 network number for testing
492 V6NET is replaced by an IPv6 network number for testing
493
494PORT_D is currently hard-wired to 1225, PORT_N to 1223, and PORT_S to 1224.
495V4NET is hardwired to 224 and V6NET to ff00. These networks are used for DNS
496testing purposes, and for testing Exim with -bh. The only requirement is that
497they are networks that can never be used for an IP address of a real host. I've
498chosen two multicast networks for the moment.
499
500If the host has no IPv6 address, "<no IPv6 address found>" is substituted but
501that does not matter because no IPv6 tests will be run. A similar substitution
502is made if there is no IPv4 address, and again, tests that actually require a
503running IPv4 interface should be skipped.
504
505If the host has more than one IPv4 or IPv6 address, the first one that
506"ifconfig" lists is used. If the only available address is 127.0.0.1 (or ::1
1b781f48 507for IPv6) it is used, but another value is preferred if available.
151b83f8
PH
508
509In situations where a specific test is not being run (for example, when setting
510up dynamic data files), TESTNUM is replaced by an empty string, but should not
511in fact occur in such files.
512
513
514HOW THE TESTS WORK
515------------------
516
517Each numbered script runs Exim (sometimes several times) with its own Exim
518configuration file. The configurations are stored in the "confs" directory,
519and before running each test, a copy of the appropriate configuration, with the
520standard substitutions, is made in the file test-config. The -C command line
521option is used to tell Exim to use this configuration.
522
523The -D option is used to pass the path of the Exim binary to the configuration.
524This is not standardly substituted, because there are two possible binaries
525that might be used in the same test (one setuid to root, the other to the exim
526user). Some tests also make use of -D to vary the configuration for different
527calls to the Exim binary.
528
529Normally, of course, Exim gives up root privilege when -C and -D are used by
530unprivileged users. We do not want this to happen when running the tests,
531because we want to be able to test all aspects of Exim, including receiving
532mail from unprivileged users. The way this is handled is as follows:
533
534At the start of the runtest script, the patchexim script is run as root. This
535script makes a copy of the Exim binary that is to be tested, patching it as it
536does so. (This is a binary patch, not a source patch.) The patch causes the
537binary, when run, to "know" that it is running in the test harness. It does not
538give up root privilege when -C and -D are used, and in a few places it takes
539other special actions, such as delaying when starting a subprocess to allow
540debug output from the parent to be written first. If you want to know more,
541grep the Exim source files for "running_in_test_harness".
542
543The patched binary is placed in the directory eximdir/exim and given the normal
544setuid root privilege. This is, of course, a dangerous binary to have lying
545around, especially if there are unprivileged users on the system. To protect
546it, the eximdir directory is created with the current user as owner, exim as
547the group owner, and with access drwx--x---. Thus, only the user who is running
548the tests (who is known to have access to root) and the exim user have access
549to the modified Exim binary. When runtest terminates, the patched binary is
550removed.
551
552Each set of tests proceeds by interpreting its controlling script. The scripts
553are in subdirectories of the "scripts" directory. They are split up according
554to the requirements of the tests they contain, with the 0000-Basic directory
555containing tests that can always be run. Run the "listtests" script to obtain a
556list of tests.
557
558
559TEST OUTPUT
560-----------
561
562Output from script runs is written to the files test-stdout and test-stderr.
563When an Exim server is involved, test-stdout-server and test-stderr-server are
564used for its output. Before being compared with the saved output, the
565non-server and server files are concatenated, so a single saved file contains
566both.
567
568A directory called spool is used for Exim's spool files, and for Exim logs.
569These locations are specified in every test's configuration file.
570
571When messages are delivered to files, the files are put in the test-mail
572directory. Output from comparisons is written to test-cf.
573
574Before comparisons are done, output texts are modified ("munged") to change or
575remove parts that are expected to vary from run to run. The modified files all
576end with the suffix "-munged". Thus, you will see test-stdout-munged,
577test-mainlog-munged, test-mail-munged, and so on. Other files whose names start
578with "test-" are created and used by some of the tests.
579
580At the end of a successful test run, the spool directory and all the files
581whose names begin with "test-" are removed. If the run ends unsuccessfully
582(typically after a "Q" response to a prompt), the spool and test files are left
583in existence so that the problem can be investigated.
584
585
586TEST COMMANDS
587-------------
588
589Each test script consists of a list of commands, each optionally preceded by
590comments (lines starting with #) and (also optionally) a line containing an
591expected return code. Some of the commands are followed by data lines
592terminated by a line of four asterisks.
593
594The first line of each script must be a comment that briefly describes the
595script. For example:
596
597 # -bS Use of HELO/RSET
598
599A line consisting just of digits is interpreted as the expected return code
600for the command that follows. The default expectation when no such line exists
601is a zero return code. For example, here is a complete test script, containing
602just one command:
603
604 # -bS Unexpected EOF in headers
605 1
606 exim -bS -odi
607 mail from:<someone@some.where>
608 rcpt to:<blackhole@HOSTNAME>
609 data
610 from: me
611 ****
612
613The expected return code in this case is 1, and the data lines are passed to
614Exim on its standard input. Both the command line and the data lines have the
615standard substitions applied to them. Thus, HOSTNAME in the example above will
616be replaced by the local host's name. Long commands can be continued over
617several lines by using \ as a continuation character. This does *not* apply to
618data lines.
619
1b781f48 620Here follows a list of supported commands. They can be divided into two groups:
151b83f8
PH
621
622
623Commands with no input
624----------------------
625
626These commands are not followed by any input data, or by a line of asterisks.
627
cfc54830 628
151b83f8
PH
629 dbmbuild <file1> <file1>
630
631This command runs the exim_dbmbuild utility to build a DBM file. It is used
632only when DBM support is available in Exim, and typically follows the use of a
633"write" command (see below) that creates the input file.
634
635
cfc54830
PH
636 dumpdb <dbname>
637
638This command runs the exim_dumpdb utility on the testing spool directory, using
639the database name given, for example: "dumpdb retry".
640
641
151b83f8
PH
642 echo <text>
643
644The text is written to the screen; this is used to output comments from
645scripts.
646
647
cfc54830
PH
648 exim_lock [options] <file name>
649
650This command runs the exim_lock utility with the given options and file name.
651The file remains locked with the following command (normally exim) is obeyed.
652
653
654 exinext <data>
655
656This command runs the exinext utility with the given argument data.
657
658
f3f065bb
PH
659 exigrep <data>
660
661This command runs the exigrep utility with the given data (the search pattern)
662on the current mainlog file.
663
664
151b83f8
PH
665 gnutls
666
667This command is present at the start of all but one of the tests that use
668GnuTLS. It copies a pre-existing parameter file into the spool directory, so
669that Exim does not have to re-create the file each time. The first GnuTLS test
cfc54830 670does not do this, in order to test that Exim can create the file.
151b83f8
PH
671
672
673 killdaemon
674
675This command must be given in any script that starts an Exim daemon, normally
676at the end. It searches for the PID file in the spool directory, and sends a
677SIGINT signal to the Exim daemon process whose PID it finds. See below for
678comments about starting Exim daemons.
679
680
681 millisleep <m>
682
683This command causes the script to sleep for m milliseconds. Nothing is output
684to the screen.
685
686
687 need_ipv4
688
689This command must be at the head of a script. If no IPv4 interface has been
690found, the entire script is skipped, and a comment is output.
691
692
693 need_ipv6
694
695This command must be at the head of a script. If no IPv6 interface has been
696found, the entire script is skipped, and a comment is output.
697
698
21c28500
PH
699 need_largefiles
700
701This command must be at the head of a script. If the Exim binary does not
702suppport large files (off_t is <= 4), the entire script is skipped, and a
703comment is output.
704
705
151b83f8
PH
706 need_move_frozen_messages
707
708This command must be at the head of a script. If the Exim binary does not have
709support for moving frozen messages (which is an optional feature), the entire
710script is skipped, and a comment is output.
711
712
713 no_message_check
714
715If this command is encountered anywhere in the script, messages that are
716delivered when the script runs are not compared with saved versions.
717
718
719 no_msglog_check
720
721If this command is encountered anywhere in the script, message log files that
722are still in existence at the end of the run (for messages that were not
723delivered) are not compared with saved versions.
724
cfc54830 725
151b83f8
PH
726 no_stderr_check
727
728If this command is encountered anywhere in the script, the stderr output from
729the run is not compared with a saved version.
730
731
732 no_stdout_check
733
734If this command is encountered anywhere in the script, the stdout output from
735the run is not compared with a saved version.
736
737
738 rmfiltertest
739
740This command indicates that the script is for a certain type of filter test, in
741which there are a lot of repetitive stdout lines that get in the way, because
742filter tests output data about the sender and recipient. Such lines are removed
743from the stdout output before comparing, for ease of human perusal.
744
745
746 sleep <n>
747
748This command causes the script to sleep for n seconds. If n is greater than
749one, "sleep <n>" is output to the screen, followed by a dot for every second
750that passes.
751
752
753 sortlog
754
755This command causes special sorting to occur on the mainlog file before
756comparison. Every sequence of contiguous delivery lines (lines containing the
757=> -> or *> flags) is sorted. This is necessary in some tests that use parallel
758deliveries because on different systems the processes may terminate in a
759different order.
760
761
cfc54830
PH
762A number of standard file management commands are also recognized. These are
763cat, chmod, chown, cp, du, ln, ls, du, mkdir, mkfifo, rm, rmdir, and touch.
764Some are run as root using "sudo".
151b83f8
PH
765
766
767Commands with input
768-------------------
769
770The remaining commands are followed by data lines for their standard input,
771terminated by four asterisks. Even if no data is required for the particular
772usage, the asterisks must be given.
773
774
775 catwrite <file name> [nxm[=start-of-line-text]]*
776
777This command operates like the "write" command, which is described below,
1b781f48 778except that the data it generates is copied to the end of the test-stdout file
151b83f8
PH
779as well as to the named file.
780
781
782
783 client [<options>] <ip address> <port> [<outgoing interface>]
784
785This command runs the auxiliary "client" program that simulates an SMTP client.
786It is controlled by a script read from its standard input, details of which are
a14e5636
PH
787given below. There are two options. One is -t, which must be followed directly
788by a number, to specify the command timeout in seconds (e.g. -t5). The default
789timeout is 1 second. The other option is -tls-on-connect, which causes the
790client to try to start up a TLS session as soon as it has connected, without
791using the STARTTLS command. The client program connects to the given IP address
792and port, using the specified interface, if one is given.
151b83f8
PH
793
794
795 client-ssl [<options>] <ip address> <port> [<outgoing interface>] \
796 [<cert file>] [<key file>]
797
798When OpenSSL is available on the host, an alternative version of the client
799program is compiled, one that supports TLS using OpenSSL. The additional
800arguments specify a certificate and key file when required. There is one
801additional option, -tls-on-connect, that causes the client to initiate TLS
802negotiation immediately on connection.
803
804
805 client-gnutls [<options>] <ip address> <port> [<outgoing interface>] \
806 [<cert file>] [<key file>]
807
808When GnuTLS is available on the host, an alternative version of the client
809program is compiled, one that supports TLS using GnuTLS. The additional
810arguments specify a certificate and key file when required. There is one
811additional option, -tls-on-connect, that causes the client to initiate TLS
812negotiation immediately on connection.
813
814
815 exim [<options>] [<arguments>]
816
817This command runs the testing version of Exim. Any occurrence of "$msg1" in the
818command line is replaced by the ID of the first (oldest) message in Exim's
819(testing) spool. "$msg2" refers to the second, and so on. The name "exim" can
820be preceded by an environment setting as in this example:
821
822 LDAPTLS_REQCERT=never exim -be
823
824It can also be preceded by a number; this specifies a number of seconds to wait
825before closing the stdout pipe to Exim, and is used for some timeout tests. For
826example:
827
828 3 exim -bs
829
830Finally, "exim" can be preceded by "sudo", to run Exim as root. If more than
831one of these prefixes is present, they must be in the above order.
832
833
834 exim_exim [<options>] [<arguments>]
835
836This runs an alternative version of Exim that is setuid to exim rather than to
837root.
838
839
840 server [<options>] <port or socket> [<connection count>]
841
842This command runs the auxiliary "server" program that simulates an SMTP (or
843other) server. It is controlled by a script that is read from its standard
844input, details of which are given below. A number of options are implemented:
845
846 -d causes the server to output debugging information
847
848 -t sets a timeout in seconds (default 5) for when the server is
849 awaiting an incoming connection
850
851 -noipv4 causes the server not to set up an IPv4 socket
852
853 -noipv6 causes the server not to set up an IPv6 socket
854
855By default, in an IPv6 environment, both kinds of socket are set up. However,
856the test script knows which interfaces actually exist on the host, and it adds
857-noipv4 or -noipv6 to the server command as required. An error occurs if both
858these options are given.
859
860The only required argument is either a port number or the path name of a Unix
861domain socket. The port is normally PORT_S, which is changed to an actual
862number by the standard substitutions. The optional final argument specifies the
863number of different connections to expect (default 1). These must happen
864serially (one at a time). There is no support for multiple simultaneous
865connections. Here are some example commands:
866
867 server PORT_S
868 server -t 10 PORT_S 3
869 server /tmp/somesocket
870
871The following lines, up to a line of four asterisks, are the server's
872controlling standard input (described below). These lines are read and
873remembered; during the following commands, until an "exim" command is reached,
874the server is run in parallel.
875
876
877 write <file name> [nxm[=start-of-line-text]]*
878
879The "write" command is a way of creating files of specific sizes for buffering
880tests, or containing specific data lines. Being able to do this from within the
881script saves holding lots of little test files. The optional argument specifies
882n lines of length m. The lines consist of the letter "a". If start of line text
883is supplied, it replaces "a"s at the start of each line. Underscores in the
884start of line text are turned into spaces. The optional argument may be
885repeated. The data lines that follow a "write" command are split into two by a
886line of four plus signs. Any above the split are written before the
887fixed-length lines, and any below the split are written after. For example:
888
889 write test-data 3x30=AB_ 1x50
890 Pre-data
891 lines
892 ++++
893 Post-data
894 lines
895 ****
896
897This command generates a file containing:
898
899 Pre-data
900 lines
901 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
902 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
903 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
904 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
905 Post-data
906 lines
907
908If there are no fixed-length line specifiers, there is no need to split the
909data, and a line of plusses is not needed.
910
911
912 [sudo] perl
913
914This command runs Perl, with the data as its standard input, to allow arbitrary
915one-off things to be done.
916
917
918CLIENT SCRIPTS
919--------------
920
921Lines in client scripts are of two kinds:
922
923(1) If a line begins with three question marks and a space, the rest of the
924 line defines the start of expected output from the server. If what is
925 received does not match, the client bombs out with an error message.
926
927(2) If a line starts with three plus signs followed by a space, the rest of the
928 line specifies a number of seconds to sleep for before proceeding.
929
930(3) Otherwise, the line is an input line line that is sent to the server. Any
931 occurrences of \r and \n in the line are turned into carriage return and
932 linefeed, respectively. This is used for testing PIPELINING.
933
934Here is a simple example:
935
936 client 127.0.0.1 PORT_D
937 ??? 250
938 EHLO xxx
939 ??? 250-
940 ??? 250
941 AUTH PLAIN AbdXi0AdnD2CVy
942 ??? 535
943 quit
944 ??? 221
945 ****
946
947In the case of client-gnutls and client-ssl, if a command is "starttls", this
948is remembered, and after a subsequent OK response, an attempt to move into TLS
949mode occurs. If a command is "starttls_wait", the client sends "starttls" but
950does not start up TLS; this is for testing timeouts. If a command is "stoptls",
951an existing TLS connection is shut down, but nothing is sent.
952
953
954SERVER SCRIPTS
955--------------
956
957The server program sleeps till a connection occurs or its timeout is reached,
958in which case it bombs out. The next set of command lines are interpreted. They
959are of the following kinds:
960
961(1) A line that starts with '>' or with a digit is an output line that is sent
962 to the client. In the case of '>':
963
964 (a) If the line starts with ">>", no terminating CRLF is sent.
965 (b) If the line starts with ">CR>", just CR is sent at the end.
966 (c) If the line starts with ">LF>", just LF is sent at the end.
967 (d) If the line starts with ">*eof", nothing is sent and the connection
968 is closed.
969
970 The data that is sent starts after the initial '>' sequence.
971
972(2) A line that starts with "*sleep" specifies a number of seconds to wait
973 before proceeding.
974
975(3) A line containing "*eof" specifies that the client is expected to close
976 the connection at this point.
977
978(4) A line containing just '.' specifies that the client is expected to send
979 many lines, terminated by one that contains just a dot.
980
981(5) Otherwise, the line defines the start of an input line that the client
982 is expected to send. To allow for lines that start with digits, the line
983 may start with '<', which is not taken as part of the input data. If the
984 input does not match, the server bombs out with an error message.
985
bbe15da8 986Here is a simple example of server use in a test script:
151b83f8
PH
987
988 server PORT_S
989 220 Greetings
990 EHLO
991 250 Hello there
992 MAIL FROM
993 250 OK
994 RCPT TO
995 250 OK
996 DATA
997 354 Send it!
998 .
999 250 OK
1000 QUIT
1001 225 OK
1002 ****
1003
1004After a "server" command in a test script, the server runs in parallel until an
1005"exim" command is reached. The "exim" command attempts to deliver one or more
1006messages to port PORT_S on the local host. When it has finished, the test
1007script waits for the "server" process to finish.
1008
bbe15da8
PH
1009The "mtpscript" program is like "server", except that it uses stdin/stdout for
1010its input and output instead of a script. However, it is not called from test
1011scripts; instead it is used as the command for pipe transports in some
1012configurations, to simulate non-socket LMTP servers.
1013
151b83f8
PH
1014
1015AUXILIARY DATA FILES
1016--------------------
1017
1018Many of the tests make use of auxiliary data files. There are two types; those
1019whose content is fixed, and those whose content needs to be varied according to
1020the current environment. The former are kept in the directory aux-fixed. The
1021latter are distributed in the directory aux-var-src, and copied with the
1022standard substitutions into the directory aux-var at the start of each test
1023run.
1024
1025Most of the auxiliary files have names that start with a test number,
1026indicating that they are specific to that one test. A few fixed files (for
1027example, some TLS certificates) are used by more than one test, and so their
1028names are not of this form.
1029
1030There are also some auxilary DNS zone files, which are described in the next
1031section.
1032
1033
1034DNS LOOKUPS AND GETHOSTBYNAME
1035-----------------------------
1036
1037The original test suite required special testing zones to be loaded into a
1038local nameserver. This is no longer a requirement for the new suite. Instead, a
1039program called fakens is used to simulate a nameserver. When Exim is running in
1040the test harness, instead of calling res_search() - the normal call to the DNS
1041resolver - it calls a testing function. This handles a few special names itself
1042(for compatibility with the old test suite), but otherwise passes the query to
1043the fakens program.
1044
1045The fakens program consults "zone files" in the directory called dnszones, and
1046returns data in the standard resource record format for Exim to process as if
1047it came from the DNS. However, if the requested domain is not in any of the
1048zones that fakens knows about, it returns a special code that causes Exim to
1049pass the query on to res_search(). The zone files are:
1050
1051 db.test.ex A zone for the domain test.ex.
1052 db.ip4.10 A zone for one special case in 10.250.0.0/16 (see below)
1053 db.ip4.V4NET A zone for the domain V4NET.in-addr.arpa.
1054 db.ip4.127 A zone for the domain 127.in-addr.arpa.
1055 db.ip6.V6NET A zone for the domain inverted(V6NET).ip6.arpa.
1056 db.ip6.0 A zone for the domain 0.ip6.arpa.
1057
1058V4NET and V6NET are substituted with the current testing networks (see above).
1059In the case of V6NET, the network is four hex digits, and it is split and
1060inverted appropriately when setting up the zone.
1061
1062These fake zone files are built dynamically from sources in the dnszones-src
1063directory by applying the standard substitutions. The test suite also builds
1064dynamic zone files for the name of the current host and its IP address(es). The
1065idea is that there should not be any need to rely on an external DNS.
1066
1067The domain names that are handled directly by Exim, without being passed to
1068fakens, are:
1069
1070 test.again.dns This always provokes a TRY_AGAIN response, for testing the
1071 handling of temporary DNS error. If the full domain name
1072 starts with digits, a delay of that many seconds occurs.
1073
1074 test.fail.dns This always provokes a NO_RECOVERY response, for testing
1075 DNS server failures.
1076
1077This special handling could now be done in the fakens program, but while the
1078old test suite is still being used it has to be done in Exim itself, so for the
1079moment it remains there.
1080
1081The use of gethostbyname() and its IPv6 friends is also subverted when Exim is
1082running in the test harness. The test code handles a few special names
1083directly; for all the others it uses DNS lookups, which are then handled as
1084just described. Thus, the use of /etc/hosts is completely bypassed. The names
1085that are specially handled are:
1086
1087 manyhome.test.ex This name is used for testing hosts with ridiculously large
1088 numbers of IP addresses; 2048 IP addresses are generated
1089 and returned. Doing it this way saves having to make the
1090 interface to fakens handle more records that can fit in the
1091 data block. The addresses that are generated are in the
1092 10.250.0.0/16 network.
1093
1094 localhost Always returns 127.0.0.1 or ::1, for IPv4 and IPv6 lookups,
1095 respectively.
1096
1097 <an IP address> If the IP address is of the correct form for the lookup
1098 type (IPv4 or IPv6), it is returned. Otherwise a panic-die
1099 error occurs.
1100
1101The reverse zone db.ip4.10 is provided just for the manyhome.test.ex case. It
1102contains a single wildcard resource record. It also contains the line
1103
1104 PASS ON NOT FOUND
1105
1106Whenever fakens finds this line in a zone file, it returns PASS_ON instead of
1107HOST_NOT_FOUND. This causes Exim to pass the query to res_search().
1108
1109****