Additional tests - didn't show any problems.
[exim.git] / test / README
CommitLineData
cfc54830 1$Cambridge: exim/test/README,v 1.5 2006/10/31 11:37:47 ph10 Exp $
151b83f8
PH
2
3EXPORTABLE EXIM TEST SUITE
4--------------------------
5
6This document last updated for:
7
cfc54830
PH
8Test Suite Version: 4.64
9Date: 31 October 2006
151b83f8
PH
10
11
12BACKGROUND
13----------
14
15For a long time, the Exim test suite was confined to Philip Hazel's
16workstation, because it relied on that particular environment. The problem is
17that an MTA such as Exim interacts a great deal with its environment, so if you
18run it somewhere else, the output will be different, which makes automatic
19checking difficult. Even in a single environment, things are not all that easy.
20For instance, if Exim delivers a message, the log line (which one would want to
21compare) contains a timestamp and an Exim message id that will be different
22each time. This issue is dealt with by a Perl script that munges the output by
23recognizing changing sequences and replacing them with fixed values before
24doing a comparison. Another problem with exporting the original test suite is
25that it assumes a version of Exim with more or less every optional feature
26enabled.
27
28This README describes a new test suite that is intended to be exportable and to
1b781f48
PH
29run in a number of different environments. The tests themselves are in no
30particular order; they accumulated over the years as Exim was extended and
31modified. They vary greatly in size and complexity. Some were specifically
32constructed to test new features; others were made to demonstrate that a bug
33had been fixed.
151b83f8
PH
34
35A few of the original tests have had to be omitted from this more general
36suite because differences in operating system behaviour make it impossible to
37generalize them. An example is a test that uses a version of Exim that is
38setuid to the Exim user rather than root, with the deliver_drop_privilege
39option set. In Linux, such a binary is able to deliver a message as the caller
40of Exim, because it can revert to the caller's uid. In FreeBSD this is not the
41case.
42
151b83f8
PH
43
44REQUIREMENTS
45------------
46
47In order to run this test suite, the following requirements must be met:
48
1b781f48 49(1) You should run the tests on a matching version of Exim, because the suite
151b83f8
PH
50 is continuously updated to test the latest features and bug fixes. The
51 version you test does not, however, have to be installed as the live
1b781f48
PH
52 version. You can of course try the tests on any version of Exim, but some
53 may fail. In particular, the test suite will fall apart horrible with
54 versions of Exim prior to 4.54.
151b83f8
PH
55
56(2) You can use any non-root login to run the tests, but there must be access
57 via "sudo" to root from this login. Privilege is required to override
58 configuration change checks and for things like cleaning up spool files,
59 but on the other hand, the tests themselves need to call Exim from a
60 non-root process. The use of "sudo" is the easiest way to achieve all this.
61 The test script uses "sudo" to do a number of things as root, so it is best
62 if you set a sudo timeout so that you do not have to keep typing a
63 password. For example, if you put
64
65 Defaults timestamp_timeout=480
66
67 in /etc/sudoers, a password lasts for 8 hours (a working day). It is
68 probably not a good idea to run the tests as the Exim user, as this is
69 recognized as special by Exim.
70
71(3) The login under which you run the tests must be in the exim group so that
72 it has access to logs, spool files, etc. The login should not be one of the
73 names "userx", "usery", "userz", or a few other simple ones such as "abcd"
1b781f48
PH
74 and "xyz" and single letters that are used in the tests. The test suite
75 expects the login to have a gecos name; I think it will now run if the
76 gecos field is empty but there may be anomalies.
151b83f8
PH
77
78(4) The directory into which you unpack the test suite must be accessible by
1b781f48
PH
79 the Exim user, so that code running as exim can access the files therein. A
80 world-readable directory is fine. However, there may be problems if the
81 path name of the directory is excessively long. This is because it
82 sometimes appears in log lines or debug output, and if it is truncated, it
151b83f8
PH
83 is no longer recognized.
84
85(5) Exim must be built with its user and group specified at build time, and
86 with certain minimum facilities, namely:
87
88 Routers: accept, dnslookup, manualroute, redirect
89 Transports: appendfile, autoreply, pipe, smtp
90 Lookups: lsearch
91
92 Most Exim binaries will have these included.
93
94(6) A C compiler is needed to build some test programs, and the test script is
95 written in Perl, so you need that.
96
97(7) Some of the tests run Exim as a daemon, and others use a testing server
98 (described below). These require TCP ports. In the configurations and
99 scripts, the ports are parameterized, but at present, fixed values are
100 written into the controlling script. These are ports 1224 to 1229. If these
101 ports are not available for use, some of the tests will fail.
102
103(8) There is an underlying assumption that the host on which the tests are
104 being run has an IPv4 address (which the test script seeks out). If there
105 is also an IPv6 address, additional tests are run when the Exim binary
106 contains IPv6 support. There are checks in the scripts for a running IPv4
107 interface; when one is not found, some tests are skipped (with a warning
108 message).
109
110
111OPTIONAL EXTRAS
112---------------
113
114If the Exim binary that is being tested contains extra functionality in
115addition to the minimum specified above, additional tests are run to exercise
116the extra functionality, except for a few special cases such as the databases
117(MySQL, PostgreSQL, LDAP) where special data is needed for the tests.
118
119
120RUNNING THE TEST SUITE
121----------------------
122
123(1) Download the tarball exim-testsuite-x.xx.tar.bz2 and unpack it, preferably
124 in a directory alongside an Exim source directory (see below).
125
126(2) cd into the exim-testsuite-x.xx directory.
127
128(3) Run "./configure" and then "make". This builds a few auxiliary programs
129 that are written in C.
130
131(4) Run "./runtest" (a Perl script) as described below.
132
133(5) If you want to see what tests are available, run "./listtests".
134
135
136BREAKING OUT OF THE TEST SCRIPT
137-------------------------------
138
139If you abandon the test run by typing ^C, the interrupt may be passed to a
140program that the script is running, or it may be passed to the script itself.
141In the former case, the script should detect that the program has ended
142abnormally. In both cases, the script tries to clean up everything, including
143killing any Exim daemons that it has started. However, there may be race
144conditions in which the clean up does not happen. If, after breaking out of a
145run, you see strange errors in the next run, look for any left-over Exim
146daemons, and kill them by hand.
147
148
149THE LISTTESTS SCRIPT
150--------------------
151
152The individual test scripts are in subdirectories of the "scripts" directory.
153If you do not supply any arguments to ./listtests, it scans all the scripts in
154all the directories, and outputs the heading line from each script. The output
155is piped through "less", and begins like this:
156
157=== 0000-Basic ===
158Basic/0001 Basic configuration setting
159Basic/0002 Common string expansions
160Basic/0003 Caseless address blocking
161...
162
163Lines that start === give the name of the subdirectory containing the test
164scripts that follow. If you supply an argument to ./listtests, it is used as a
165Perl pattern to match case-independently against the names of the
166subdirectories. Only those that match are scanned. For example, "./listtests
167ipv6" outputs this:
168
169=== 1000-Basic-ipv6 ===
170=== Requires: support IPv6
171Basic-ipv6/1000 -bh and non-canonical IPv6 addresses
172Basic-ipv6/1001 recognizing IPv6 address in HELO/EHLO
173
174=== 2250-dnsdb-ipv6 ===
175=== Requires: support IPv6
176 lookup dnsdb
177dnsdb-ipv6/2250 dnsdb ipv6 lookup in string expansions
178
179If you supply a second argument to ./listtests, it is used as a Perl pattern to
180match case-independently against the individual script titles. For example,
181"./listtests . mx" lists all tests whose titles contain "mx", because "."
182matches all the subdirectory names.
183
184
185THE RUNTEST SCRIPT
186------------------
187
188If you do not supply any arguments to ./runtest, it searches for an Exim
189source tree at the same level as the test suite directory. It then looks for an
190Exim binary in a "build" directory of that source tree. If there are several
191Exim source trees, it chooses the latest version of Exim. Consider the
192following example:
193
194 $ ls -F /source/exim
1b781f48 195 exim-4.60/ exim-4.62/ exim-testsuite-x.xx/
151b83f8 196
1b781f48
PH
197A simple ./runtest from within the test suite will use a 4.62 binary if it
198finds one, otherwise a 4.60 binary. If a binary cannot be found, the script
151b83f8
PH
199prompts for one. Alternatively, you can supply the binary on the command line:
200
201 ./runtest /usr/exim/bin/exim
202
1b781f48
PH
203A matching test suite is released with each Exim release; if you use a test
204suite that does not match the binary, some tests may fail.
205
206The test suite uses some of the Exim utilities (such as exim_dbmbuild), and it
207expects to find them in the same directory as Exim itself. If they are not
208found, the tests that use them are omitted. A suitable comment is output.
151b83f8
PH
209
210On the ./runtest command line, following the name of the binary, if present,
211there may be a number of options and then one or two numbers. The full syntax
212is as follows:
213
214 ./runtest [binary name] [runtest options] [exim options] \
215 [first test] [last test]
216
217There are some options for the ./runtest script itself:
218
219 -DEBUG This option is for debugging the test script. It causes some
220 tracing information to be output.
221
222 -DIFF By default, file comparisons are done using a private compare
223 command called "cf", which is built from source that is provided in
224 the src directory. This is a command I've had for nearly 20 years -
225 look at the source comments for its history - whose output I
226 prefer. However, if you want to use "diff" instead, give -DIFF as a
227 runtest option. In that case, "diff -u" is used for comparisons.
228 (If it turns out that most people prefer to use diff, I'll change
229 the default.)
230
231 -KEEP Normally, after a successful run, the test output files are
232 deleted. This option prevents this. It is useful when running a
233 single test, in order to look at the actual output before it is
234 modified for comparison with saved output.
235
236 -NOIPV4 Pretend that an IPv4 interface was not found. This is useful for
237 testing that the test suite correctly skips tests that require
238 a running IPv4 interface.
239
240 -NOIPV6 Pretend that an IPv6 interface was not found. This is useful for
241 testing that the test suite correctly skips tests that require
242 a running IPv6 interface.
243
244 -UPDATE If this option is set, any detected changes in test output are
245 automatically accepted and used to update the stored copies of the
246 output. It is a dangerous option, but it useful for the test suite
247 maintainer after making a change to the code that affects a lot of
248 tests (for example, the wording of a message).
249
250The options for ./runtest must be given first (but after the name of the
251binary, if present). Any further options, that is, items on the command line
252that start with a hyphen, are passed to the Exim binary when it is run as part
253of a test. The only sensible use of this is to pass "-d" in order to run a test
254with debugging enabled. Any other options are likely to conflict with options
255that are set in the tests. Some tests are already set up to run with debugging.
256In these cases, -d on the command line overrides their own debug settings.
257
258The final two arguments specify the range of tests to be run. Test numbers lie
259in the range 1 to 9999. If no numbers are given, the defaults are 1 and 8999
260(sic). Tests with higher numbers (9000 upwards) are not run automatically
261because they require specific data (such as a particular MySQL table) that is
262unlikely to be generally available.
263
264Tests that require certain optional features of Exim are grouped by number, so
265in any given range, not all the tests will exist. Non-existent tests are just
266skipped, but if there are no tests at all in the given range, a message is
267output.
268
269If you give only one number, just that test is run (if it exists). Instead of a
270second number, you can give the character "+", which is interpreted as "to the
271end". Normally this is 8999; if the starting number is 9000 or higher, "+" is
272interpreted as 9999. Examples:
273
274 ./runtest 1300
275 ./runtest 1400 1699
276 ./runtest /usr/sbin/exim 5000 +
277 ./runtest -DIFF -d 81
278
279When the script starts up, the first thing it does is to check that you have
280sudo access to root. Then it outputs the version number of the Exim binary that
281it is testing, and also information about the optional facilities that are
282present (obtained from "exim -bV"). This is followed by some environmental
283information, including the current login id and the hosts's IP address. The
284script checks that the current user is in the Exim group, and that the Exim
285user has access to the test suite directory.
286
287The script outputs the list of tests requested, and a list of tests that will
288be omitted because the relevant optional facilities are not in the binary. You
289are then invited to press Return to start the tests running.
290
291
292TEST OUTPUT
293-----------
294
295When all goes well, the only permanent output is the identity of the tests as
296they are run, and "Script completed" for each test script, for example:
297
298 Basic/0001 Basic configuration setting
299 Script completed
300 Basic/0002 Basic string expansions
301 Script completed
302 Basic/0003 Caseless address blocking
303 Script completed
304 Basic/0004 Caseful address blocking
305 Script completed
306 Basic/0005 -bs to simple local delivery
307 ...
308
309While a script is running, it shows "Test n" on the screen, for each of the
310Exim tests within the script. There may also be comments from some tests when a
311delay is expected, for example, if there is a "sleep" while testing a timeout.
312
313Before each set of optional tests, an extra identifying line is output. For
314example:
315
316 >>> The following tests require: authenticator cram_md5
317 CRAM-MD5/2500 CRAM-MD5 server tests
318 Script completed
319 CRAM-MD5/2501 CRAM-MD5 client tests
320 Script completed
321
322If a test fails, you are shown the output of the text comparison that failed,
323and prompted as to what to do next. The output is shown using the "less"
1b781f48
PH
324command, or "more" if "less" is not available. The options for "less" are set
325to that it automatically exits if there is less that a screenful of output. By
326default, the output is from the "cf" program, and might look like this:
151b83f8
PH
327
328 DBM/1300 DBM files and exim_dbmbuild
329 ===============
330 Lines 7-9 of "test-stdout-munged" do not match lines 7-11 of "stdout/1300".
331 ----------
332 exim_dbmbuild exit code = 1
333 Continued set of lines is too long: max permitted length is 99999
334 exim_dbmbuild exit code = 1
335 ----------
336 dbmbuild abandoned
337 exim_dbmbuild exit code = 2
338 Continued set of lines is too long: max permitted length is 99999
339 dbmbuild abandoned
340 exim_dbmbuild exit code = 2
341 ===============
342 1 difference found.
343 "test-stdout-munged" contains 16 lines; "stdout/1300" contains 18 lines.
344
345 Continue, Update & retry, Quit? [Q]
346
347This example was generated by running the test with a version of Exim
348that had a bug in the exim_dbmbuild utility (the bug was fixed at release
3494.53). See "How the tests work" below for a description of the files that are
350used. In this case, the standard output differed from what was expected.
351
352The reply to the prompt must either be empty, in which case it takes the
353default that is given in brackets (in this case Q), or a single letter, in
354upper or lower case (in this case, one of C, U, or Q). If you type anything
355else, the prompt is repeated.
356
357"Continue" carries on as if the files had matched; that is, it ignores the
358mismatch. Any other output files for the same test will be compared before
359moving on to the next test.
360
361"Update & retry" copies the new file to the saved file, and reruns the test
362after doing any further comparisons that may be necessary.
363
364Other circumstances give rise to other prompts. If a test generates output for
365which there is no saved data, the prompt (after a message stating which file is
366unexpectely not empty) is:
367
368 Continue, Show, or Quit? [Q]
369
370"Show" displays the data on the screen, and then you get the "Continue..."
371prompt. If a test ends with an unexpected return code, the prompt is:
372
373 show stdErr, show stdOut, Continue (without file comparison), or Quit? [Q]
374
375Typically in these cases there will be something interesting in the stderr
376or stdout output. There is a similar prompt after the "server" auxiliary
377program fails.
378
379
380OPENSSL AND GNUTLS ERROR MESSAGES
381---------------------------------
382
383Some of the TLS tests deliberately cause errors to check how Exim handles them.
384It has been observed that different releases of the OpenSSL and GnuTLS
385libraries generate different error messages. This may cause the comparison with
386the saved output to fail. Such errors can be ignored.
387
388
1b781f48
PH
389OTHER ISSUES
390------------
391
392. Some of the tests are time-sensitive (e.g. when testing timeouts, as in test
393 461). These may fail if run on a host that is also running a lot of other
394 processes.
395
396. Some versions of "ls" use a different format for times and dates. This can
397 cause test 345 to fail.
398
399. Test 0142 tests open file descriptors; on some hosts the output may vary.
400
401
151b83f8
PH
402OTHER SCRIPTS AND PROGRAMS
403--------------------------
404
405There is a freestanding Perl script called "listtests" that scans the test
406scripts and outputs a list of all the tests, with a short descriptive comment
407for each one. Special requirements for groups of tests are also noted.
408
409The main runtest script makes use of a second Perl script and some compiled C
410programs. These are:
411
412patchexim A Perl script that makes a patched version of Exim (see the
413 next section for details).
414
415bin/cf A text comparison program (see above).
416
417bin/checkaccess A program that is run as root; it changes uid/gid to the
418 Exim user and group, and then checks that it can access
419 files in the test suite's directory.
420
421bin/client A script-driven SMTP client simulation.
422
423bin/client-gnutls A script-driven SMTP client simulation with GnuTLS support.
424 This is built only if GnuTLS support is detected on the host.
425
426bin/client-ssl A script-driven SMTP client simulation with OpenSSL support.
427 This is built only if OpenSSL support is detected on the
428 host.
429
430bin/fakens A fake "nameserver" for DNS tests (see below for details).
431
432bin/fd A program that outputs details of open file descriptors.
433
434bin/iefbr14 A program that does nothing, and returns 0. It's just like
435 the "true" command, but it is in a known place.
436
437bin/loaded Some dynamically loaded functions for testing dlfunc support.
438
439bin/server A script-driven SMTP server simulation.
440
441The runtest script also makes use of a number of ordinary commands such as
442"cp", "kill", "more", and "rm", via the system() call. In some cases these are
443run as root by means of sudo.
444
445
446STANDARD SUBSTITUTIONS
447----------------------
448
449In the following sections, there are several references to the "standard
450substitutions". These make changes to some of the stored files when they are
451used in a test. To save repetition, the substitutions themselves are documented
452here:
453
454 CALLER is replaced by the login name of the user running the tests
1b781f48 455 CALLERGROUP is replaced by the caller's group id
151b83f8
PH
456 CALLER_GID is replaced by the caller's group id
457 CALLER_UID is replaced by the caller's user id
458 DIR is replaced by the name of the test-suite directory
459 EXIMGROUP is replaced by the name of the Exim group
460 EXIMUSER is replaced by the name of the Exim user
461 HOSTIPV4 is replaced by the local host's IPv4 address
462 HOSTIPV6 is replaced by the local host's IPv6 address
463 HOSTNAME is replaced by the local host's name
464 PORT_D is replaced by a port number for normal daemon use
465 PORT_N is replaced by a port number that should never respond
466 PORT_S is replaced by a port number for normal bin/server use
467 TESTNUM is replaced by the current test number
468 V4NET is replaced by an IPv4 network number for testing
469 V6NET is replaced by an IPv6 network number for testing
470
471PORT_D is currently hard-wired to 1225, PORT_N to 1223, and PORT_S to 1224.
472V4NET is hardwired to 224 and V6NET to ff00. These networks are used for DNS
473testing purposes, and for testing Exim with -bh. The only requirement is that
474they are networks that can never be used for an IP address of a real host. I've
475chosen two multicast networks for the moment.
476
477If the host has no IPv6 address, "<no IPv6 address found>" is substituted but
478that does not matter because no IPv6 tests will be run. A similar substitution
479is made if there is no IPv4 address, and again, tests that actually require a
480running IPv4 interface should be skipped.
481
482If the host has more than one IPv4 or IPv6 address, the first one that
483"ifconfig" lists is used. If the only available address is 127.0.0.1 (or ::1
1b781f48 484for IPv6) it is used, but another value is preferred if available.
151b83f8
PH
485
486In situations where a specific test is not being run (for example, when setting
487up dynamic data files), TESTNUM is replaced by an empty string, but should not
488in fact occur in such files.
489
490
491HOW THE TESTS WORK
492------------------
493
494Each numbered script runs Exim (sometimes several times) with its own Exim
495configuration file. The configurations are stored in the "confs" directory,
496and before running each test, a copy of the appropriate configuration, with the
497standard substitutions, is made in the file test-config. The -C command line
498option is used to tell Exim to use this configuration.
499
500The -D option is used to pass the path of the Exim binary to the configuration.
501This is not standardly substituted, because there are two possible binaries
502that might be used in the same test (one setuid to root, the other to the exim
503user). Some tests also make use of -D to vary the configuration for different
504calls to the Exim binary.
505
506Normally, of course, Exim gives up root privilege when -C and -D are used by
507unprivileged users. We do not want this to happen when running the tests,
508because we want to be able to test all aspects of Exim, including receiving
509mail from unprivileged users. The way this is handled is as follows:
510
511At the start of the runtest script, the patchexim script is run as root. This
512script makes a copy of the Exim binary that is to be tested, patching it as it
513does so. (This is a binary patch, not a source patch.) The patch causes the
514binary, when run, to "know" that it is running in the test harness. It does not
515give up root privilege when -C and -D are used, and in a few places it takes
516other special actions, such as delaying when starting a subprocess to allow
517debug output from the parent to be written first. If you want to know more,
518grep the Exim source files for "running_in_test_harness".
519
520The patched binary is placed in the directory eximdir/exim and given the normal
521setuid root privilege. This is, of course, a dangerous binary to have lying
522around, especially if there are unprivileged users on the system. To protect
523it, the eximdir directory is created with the current user as owner, exim as
524the group owner, and with access drwx--x---. Thus, only the user who is running
525the tests (who is known to have access to root) and the exim user have access
526to the modified Exim binary. When runtest terminates, the patched binary is
527removed.
528
529Each set of tests proceeds by interpreting its controlling script. The scripts
530are in subdirectories of the "scripts" directory. They are split up according
531to the requirements of the tests they contain, with the 0000-Basic directory
532containing tests that can always be run. Run the "listtests" script to obtain a
533list of tests.
534
535
536TEST OUTPUT
537-----------
538
539Output from script runs is written to the files test-stdout and test-stderr.
540When an Exim server is involved, test-stdout-server and test-stderr-server are
541used for its output. Before being compared with the saved output, the
542non-server and server files are concatenated, so a single saved file contains
543both.
544
545A directory called spool is used for Exim's spool files, and for Exim logs.
546These locations are specified in every test's configuration file.
547
548When messages are delivered to files, the files are put in the test-mail
549directory. Output from comparisons is written to test-cf.
550
551Before comparisons are done, output texts are modified ("munged") to change or
552remove parts that are expected to vary from run to run. The modified files all
553end with the suffix "-munged". Thus, you will see test-stdout-munged,
554test-mainlog-munged, test-mail-munged, and so on. Other files whose names start
555with "test-" are created and used by some of the tests.
556
557At the end of a successful test run, the spool directory and all the files
558whose names begin with "test-" are removed. If the run ends unsuccessfully
559(typically after a "Q" response to a prompt), the spool and test files are left
560in existence so that the problem can be investigated.
561
562
563TEST COMMANDS
564-------------
565
566Each test script consists of a list of commands, each optionally preceded by
567comments (lines starting with #) and (also optionally) a line containing an
568expected return code. Some of the commands are followed by data lines
569terminated by a line of four asterisks.
570
571The first line of each script must be a comment that briefly describes the
572script. For example:
573
574 # -bS Use of HELO/RSET
575
576A line consisting just of digits is interpreted as the expected return code
577for the command that follows. The default expectation when no such line exists
578is a zero return code. For example, here is a complete test script, containing
579just one command:
580
581 # -bS Unexpected EOF in headers
582 1
583 exim -bS -odi
584 mail from:<someone@some.where>
585 rcpt to:<blackhole@HOSTNAME>
586 data
587 from: me
588 ****
589
590The expected return code in this case is 1, and the data lines are passed to
591Exim on its standard input. Both the command line and the data lines have the
592standard substitions applied to them. Thus, HOSTNAME in the example above will
593be replaced by the local host's name. Long commands can be continued over
594several lines by using \ as a continuation character. This does *not* apply to
595data lines.
596
1b781f48 597Here follows a list of supported commands. They can be divided into two groups:
151b83f8
PH
598
599
600Commands with no input
601----------------------
602
603These commands are not followed by any input data, or by a line of asterisks.
604
cfc54830 605
151b83f8
PH
606 dbmbuild <file1> <file1>
607
608This command runs the exim_dbmbuild utility to build a DBM file. It is used
609only when DBM support is available in Exim, and typically follows the use of a
610"write" command (see below) that creates the input file.
611
612
cfc54830
PH
613 dumpdb <dbname>
614
615This command runs the exim_dumpdb utility on the testing spool directory, using
616the database name given, for example: "dumpdb retry".
617
618
151b83f8
PH
619 echo <text>
620
621The text is written to the screen; this is used to output comments from
622scripts.
623
624
cfc54830
PH
625 exim_lock [options] <file name>
626
627This command runs the exim_lock utility with the given options and file name.
628The file remains locked with the following command (normally exim) is obeyed.
629
630
631 exinext <data>
632
633This command runs the exinext utility with the given argument data.
634
635
151b83f8
PH
636 gnutls
637
638This command is present at the start of all but one of the tests that use
639GnuTLS. It copies a pre-existing parameter file into the spool directory, so
640that Exim does not have to re-create the file each time. The first GnuTLS test
cfc54830 641does not do this, in order to test that Exim can create the file.
151b83f8
PH
642
643
644 killdaemon
645
646This command must be given in any script that starts an Exim daemon, normally
647at the end. It searches for the PID file in the spool directory, and sends a
648SIGINT signal to the Exim daemon process whose PID it finds. See below for
649comments about starting Exim daemons.
650
651
652 millisleep <m>
653
654This command causes the script to sleep for m milliseconds. Nothing is output
655to the screen.
656
657
658 need_ipv4
659
660This command must be at the head of a script. If no IPv4 interface has been
661found, the entire script is skipped, and a comment is output.
662
663
664 need_ipv6
665
666This command must be at the head of a script. If no IPv6 interface has been
667found, the entire script is skipped, and a comment is output.
668
669
21c28500
PH
670 need_largefiles
671
672This command must be at the head of a script. If the Exim binary does not
673suppport large files (off_t is <= 4), the entire script is skipped, and a
674comment is output.
675
676
151b83f8
PH
677 need_move_frozen_messages
678
679This command must be at the head of a script. If the Exim binary does not have
680support for moving frozen messages (which is an optional feature), the entire
681script is skipped, and a comment is output.
682
683
684 no_message_check
685
686If this command is encountered anywhere in the script, messages that are
687delivered when the script runs are not compared with saved versions.
688
689
690 no_msglog_check
691
692If this command is encountered anywhere in the script, message log files that
693are still in existence at the end of the run (for messages that were not
694delivered) are not compared with saved versions.
695
cfc54830 696
151b83f8
PH
697 no_stderr_check
698
699If this command is encountered anywhere in the script, the stderr output from
700the run is not compared with a saved version.
701
702
703 no_stdout_check
704
705If this command is encountered anywhere in the script, the stdout output from
706the run is not compared with a saved version.
707
708
709 rmfiltertest
710
711This command indicates that the script is for a certain type of filter test, in
712which there are a lot of repetitive stdout lines that get in the way, because
713filter tests output data about the sender and recipient. Such lines are removed
714from the stdout output before comparing, for ease of human perusal.
715
716
717 sleep <n>
718
719This command causes the script to sleep for n seconds. If n is greater than
720one, "sleep <n>" is output to the screen, followed by a dot for every second
721that passes.
722
723
724 sortlog
725
726This command causes special sorting to occur on the mainlog file before
727comparison. Every sequence of contiguous delivery lines (lines containing the
728=> -> or *> flags) is sorted. This is necessary in some tests that use parallel
729deliveries because on different systems the processes may terminate in a
730different order.
731
732
cfc54830
PH
733A number of standard file management commands are also recognized. These are
734cat, chmod, chown, cp, du, ln, ls, du, mkdir, mkfifo, rm, rmdir, and touch.
735Some are run as root using "sudo".
151b83f8
PH
736
737
738Commands with input
739-------------------
740
741The remaining commands are followed by data lines for their standard input,
742terminated by four asterisks. Even if no data is required for the particular
743usage, the asterisks must be given.
744
745
746 catwrite <file name> [nxm[=start-of-line-text]]*
747
748This command operates like the "write" command, which is described below,
1b781f48 749except that the data it generates is copied to the end of the test-stdout file
151b83f8
PH
750as well as to the named file.
751
752
753
754 client [<options>] <ip address> <port> [<outgoing interface>]
755
756This command runs the auxiliary "client" program that simulates an SMTP client.
757It is controlled by a script read from its standard input, details of which are
758given below. The only option is -t, which must be followed by a number, to
759specify the command timeout in seconds. The program connects to the given IP
760address and port, using the specified interface, if one is given.
761
762
763 client-ssl [<options>] <ip address> <port> [<outgoing interface>] \
764 [<cert file>] [<key file>]
765
766When OpenSSL is available on the host, an alternative version of the client
767program is compiled, one that supports TLS using OpenSSL. The additional
768arguments specify a certificate and key file when required. There is one
769additional option, -tls-on-connect, that causes the client to initiate TLS
770negotiation immediately on connection.
771
772
773 client-gnutls [<options>] <ip address> <port> [<outgoing interface>] \
774 [<cert file>] [<key file>]
775
776When GnuTLS is available on the host, an alternative version of the client
777program is compiled, one that supports TLS using GnuTLS. The additional
778arguments specify a certificate and key file when required. There is one
779additional option, -tls-on-connect, that causes the client to initiate TLS
780negotiation immediately on connection.
781
782
783 exim [<options>] [<arguments>]
784
785This command runs the testing version of Exim. Any occurrence of "$msg1" in the
786command line is replaced by the ID of the first (oldest) message in Exim's
787(testing) spool. "$msg2" refers to the second, and so on. The name "exim" can
788be preceded by an environment setting as in this example:
789
790 LDAPTLS_REQCERT=never exim -be
791
792It can also be preceded by a number; this specifies a number of seconds to wait
793before closing the stdout pipe to Exim, and is used for some timeout tests. For
794example:
795
796 3 exim -bs
797
798Finally, "exim" can be preceded by "sudo", to run Exim as root. If more than
799one of these prefixes is present, they must be in the above order.
800
801
802 exim_exim [<options>] [<arguments>]
803
804This runs an alternative version of Exim that is setuid to exim rather than to
805root.
806
807
808 server [<options>] <port or socket> [<connection count>]
809
810This command runs the auxiliary "server" program that simulates an SMTP (or
811other) server. It is controlled by a script that is read from its standard
812input, details of which are given below. A number of options are implemented:
813
814 -d causes the server to output debugging information
815
816 -t sets a timeout in seconds (default 5) for when the server is
817 awaiting an incoming connection
818
819 -noipv4 causes the server not to set up an IPv4 socket
820
821 -noipv6 causes the server not to set up an IPv6 socket
822
823By default, in an IPv6 environment, both kinds of socket are set up. However,
824the test script knows which interfaces actually exist on the host, and it adds
825-noipv4 or -noipv6 to the server command as required. An error occurs if both
826these options are given.
827
828The only required argument is either a port number or the path name of a Unix
829domain socket. The port is normally PORT_S, which is changed to an actual
830number by the standard substitutions. The optional final argument specifies the
831number of different connections to expect (default 1). These must happen
832serially (one at a time). There is no support for multiple simultaneous
833connections. Here are some example commands:
834
835 server PORT_S
836 server -t 10 PORT_S 3
837 server /tmp/somesocket
838
839The following lines, up to a line of four asterisks, are the server's
840controlling standard input (described below). These lines are read and
841remembered; during the following commands, until an "exim" command is reached,
842the server is run in parallel.
843
844
845 write <file name> [nxm[=start-of-line-text]]*
846
847The "write" command is a way of creating files of specific sizes for buffering
848tests, or containing specific data lines. Being able to do this from within the
849script saves holding lots of little test files. The optional argument specifies
850n lines of length m. The lines consist of the letter "a". If start of line text
851is supplied, it replaces "a"s at the start of each line. Underscores in the
852start of line text are turned into spaces. The optional argument may be
853repeated. The data lines that follow a "write" command are split into two by a
854line of four plus signs. Any above the split are written before the
855fixed-length lines, and any below the split are written after. For example:
856
857 write test-data 3x30=AB_ 1x50
858 Pre-data
859 lines
860 ++++
861 Post-data
862 lines
863 ****
864
865This command generates a file containing:
866
867 Pre-data
868 lines
869 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
870 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
871 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
872 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
873 Post-data
874 lines
875
876If there are no fixed-length line specifiers, there is no need to split the
877data, and a line of plusses is not needed.
878
879
880 [sudo] perl
881
882This command runs Perl, with the data as its standard input, to allow arbitrary
883one-off things to be done.
884
885
886CLIENT SCRIPTS
887--------------
888
889Lines in client scripts are of two kinds:
890
891(1) If a line begins with three question marks and a space, the rest of the
892 line defines the start of expected output from the server. If what is
893 received does not match, the client bombs out with an error message.
894
895(2) If a line starts with three plus signs followed by a space, the rest of the
896 line specifies a number of seconds to sleep for before proceeding.
897
898(3) Otherwise, the line is an input line line that is sent to the server. Any
899 occurrences of \r and \n in the line are turned into carriage return and
900 linefeed, respectively. This is used for testing PIPELINING.
901
902Here is a simple example:
903
904 client 127.0.0.1 PORT_D
905 ??? 250
906 EHLO xxx
907 ??? 250-
908 ??? 250
909 AUTH PLAIN AbdXi0AdnD2CVy
910 ??? 535
911 quit
912 ??? 221
913 ****
914
915In the case of client-gnutls and client-ssl, if a command is "starttls", this
916is remembered, and after a subsequent OK response, an attempt to move into TLS
917mode occurs. If a command is "starttls_wait", the client sends "starttls" but
918does not start up TLS; this is for testing timeouts. If a command is "stoptls",
919an existing TLS connection is shut down, but nothing is sent.
920
921
922SERVER SCRIPTS
923--------------
924
925The server program sleeps till a connection occurs or its timeout is reached,
926in which case it bombs out. The next set of command lines are interpreted. They
927are of the following kinds:
928
929(1) A line that starts with '>' or with a digit is an output line that is sent
930 to the client. In the case of '>':
931
932 (a) If the line starts with ">>", no terminating CRLF is sent.
933 (b) If the line starts with ">CR>", just CR is sent at the end.
934 (c) If the line starts with ">LF>", just LF is sent at the end.
935 (d) If the line starts with ">*eof", nothing is sent and the connection
936 is closed.
937
938 The data that is sent starts after the initial '>' sequence.
939
940(2) A line that starts with "*sleep" specifies a number of seconds to wait
941 before proceeding.
942
943(3) A line containing "*eof" specifies that the client is expected to close
944 the connection at this point.
945
946(4) A line containing just '.' specifies that the client is expected to send
947 many lines, terminated by one that contains just a dot.
948
949(5) Otherwise, the line defines the start of an input line that the client
950 is expected to send. To allow for lines that start with digits, the line
951 may start with '<', which is not taken as part of the input data. If the
952 input does not match, the server bombs out with an error message.
953
954Here is a simple server example:
955
956 server PORT_S
957 220 Greetings
958 EHLO
959 250 Hello there
960 MAIL FROM
961 250 OK
962 RCPT TO
963 250 OK
964 DATA
965 354 Send it!
966 .
967 250 OK
968 QUIT
969 225 OK
970 ****
971
972After a "server" command in a test script, the server runs in parallel until an
973"exim" command is reached. The "exim" command attempts to deliver one or more
974messages to port PORT_S on the local host. When it has finished, the test
975script waits for the "server" process to finish.
976
977
978AUXILIARY DATA FILES
979--------------------
980
981Many of the tests make use of auxiliary data files. There are two types; those
982whose content is fixed, and those whose content needs to be varied according to
983the current environment. The former are kept in the directory aux-fixed. The
984latter are distributed in the directory aux-var-src, and copied with the
985standard substitutions into the directory aux-var at the start of each test
986run.
987
988Most of the auxiliary files have names that start with a test number,
989indicating that they are specific to that one test. A few fixed files (for
990example, some TLS certificates) are used by more than one test, and so their
991names are not of this form.
992
993There are also some auxilary DNS zone files, which are described in the next
994section.
995
996
997DNS LOOKUPS AND GETHOSTBYNAME
998-----------------------------
999
1000The original test suite required special testing zones to be loaded into a
1001local nameserver. This is no longer a requirement for the new suite. Instead, a
1002program called fakens is used to simulate a nameserver. When Exim is running in
1003the test harness, instead of calling res_search() - the normal call to the DNS
1004resolver - it calls a testing function. This handles a few special names itself
1005(for compatibility with the old test suite), but otherwise passes the query to
1006the fakens program.
1007
1008The fakens program consults "zone files" in the directory called dnszones, and
1009returns data in the standard resource record format for Exim to process as if
1010it came from the DNS. However, if the requested domain is not in any of the
1011zones that fakens knows about, it returns a special code that causes Exim to
1012pass the query on to res_search(). The zone files are:
1013
1014 db.test.ex A zone for the domain test.ex.
1015 db.ip4.10 A zone for one special case in 10.250.0.0/16 (see below)
1016 db.ip4.V4NET A zone for the domain V4NET.in-addr.arpa.
1017 db.ip4.127 A zone for the domain 127.in-addr.arpa.
1018 db.ip6.V6NET A zone for the domain inverted(V6NET).ip6.arpa.
1019 db.ip6.0 A zone for the domain 0.ip6.arpa.
1020
1021V4NET and V6NET are substituted with the current testing networks (see above).
1022In the case of V6NET, the network is four hex digits, and it is split and
1023inverted appropriately when setting up the zone.
1024
1025These fake zone files are built dynamically from sources in the dnszones-src
1026directory by applying the standard substitutions. The test suite also builds
1027dynamic zone files for the name of the current host and its IP address(es). The
1028idea is that there should not be any need to rely on an external DNS.
1029
1030The domain names that are handled directly by Exim, without being passed to
1031fakens, are:
1032
1033 test.again.dns This always provokes a TRY_AGAIN response, for testing the
1034 handling of temporary DNS error. If the full domain name
1035 starts with digits, a delay of that many seconds occurs.
1036
1037 test.fail.dns This always provokes a NO_RECOVERY response, for testing
1038 DNS server failures.
1039
1040This special handling could now be done in the fakens program, but while the
1041old test suite is still being used it has to be done in Exim itself, so for the
1042moment it remains there.
1043
1044The use of gethostbyname() and its IPv6 friends is also subverted when Exim is
1045running in the test harness. The test code handles a few special names
1046directly; for all the others it uses DNS lookups, which are then handled as
1047just described. Thus, the use of /etc/hosts is completely bypassed. The names
1048that are specially handled are:
1049
1050 manyhome.test.ex This name is used for testing hosts with ridiculously large
1051 numbers of IP addresses; 2048 IP addresses are generated
1052 and returned. Doing it this way saves having to make the
1053 interface to fakens handle more records that can fit in the
1054 data block. The addresses that are generated are in the
1055 10.250.0.0/16 network.
1056
1057 localhost Always returns 127.0.0.1 or ::1, for IPv4 and IPv6 lookups,
1058 respectively.
1059
1060 <an IP address> If the IP address is of the correct form for the lookup
1061 type (IPv4 or IPv6), it is returned. Otherwise a panic-die
1062 error occurs.
1063
1064The reverse zone db.ip4.10 is provided just for the manyhome.test.ex case. It
1065contains a single wildcard resource record. It also contains the line
1066
1067 PASS ON NOT FOUND
1068
1069Whenever fakens finds this line in a zone file, it returns PASS_ON instead of
1070HOST_NOT_FOUND. This causes Exim to pass the query to res_search().
1071
1072****