[exim.git] / doc / doc-txt / test-harness.txt

$Cambridge: exim/doc/doc-txt/test-harness.txt,v 1.1 2006/02/16 10:05:33 ph10 Exp $

EXIM'S BEHAVIOUR CHANGES WHEN RUNNING IN THE TEST HARNESS
---------------------------------------------------------

When Exim is running in its test harness, via the scripts in the exim-testsuite
distribution, its behaviour is altered in a few ways, in order to make the
regression testing work properly. The documentation for the test suite
describes how a copy of the Exim binary is taken and patched in order to get it
to run in the test harness. This document briefly lists the behavioural changes
that result. They come into play when the Boolean variable running_in_test_
harness is true.


Privilege
---------

Exim does not give up its root privilege when called with -C or -D, nor does it
insist on the caller being an admin user when starting a daemon, a queue
runner, or requesting debug output.


Small Pauses
------------

In a number of places, typically when a subprocess has been forked, there are
short pauses of half or one second in one of the processes. This allows the
other process to "go first"; it ensures that debugging or logging output always
appears in the same order.


Daemon
------

The daemon always writes a pid file when running in the test harness.


CRAM-MD5
--------

The cram_md5 authenticator always uses the same challenge string.


Appendfile
----------

After a quota error, the "time since last read" for the file is forced to 10s,
for repeatability.


Memory management
-----------------

Memory management debugging output contains only the store pool and the size
(other information is too variable). New memory is initialized to contain F0 in
all bytes.


Queue running
-------------

There's a facility (-Tqt) for fudging queue times for testing retry logic.


Syslog
------

Exim never writes to syslog in the test harness. Attempts to do so are silently
ignored. None of the tests actually specify syslog logging for any actual log
lines, but there is one that tests the inability to open the main and panic
logs, which by default then tries to write to syslog.


SMTP connection timeout
-----------------------

In order to be able to test timeout handling, a "connection refused" error is
converted into a timeout if the timeout value is set to 999999s.


Random numbers
--------------

The seed for the pseudo-random number generator is set to a fixed value in the
test harness, to ensure repeatability.


Bounce messages
---------------

When Exim is submitting a bounce message to itself, unless the configuration
has set queue_only, it uses -odi so that the bounce is delivered before the
subprocess returns. This avoids a race that might put log lines in an arbitrary
order.


DNS lookups
-----------

The real DNS resolver is never called. Instead, a fake resolver, which runs as
a separate program, is used. It is part of the test suite and is documented
there. This ensures complete control over the exact results of any DNS lookups.

An attempt to look up a PTR record for 99.99.99.99 or an IP address for a host
whose name ends with .test.again.dns always yields a "try again" error.

A fake function is called instead of gethostbyname(). It recognizes the name
"manyhome.test.ex" and generates a humungous number of IP addresses. It also
recognizes an unqualified "localhost" and forces it to the appropriate loopback
address (IPv4 or IPv6, as required). IP addresses are treated as literals. For
other names, it does a DNS lookup (which of course actually calls the fake
resolver) to find the host name.


User names
----------

If unknown_login is set, it forces the login name, thus overriding the actual
login for the test suite caller. When this happens, unknown_username provides a
user name if it is set; otherwise an empty string is used.


Ident
-----

If -bh is used and both the sending host port and the incoming interface port
are supplied, an ident (RFC 1413) call is made for testing purposes.


Debug output
------------

Debugging output from the function that waits for the clock to tick at an
appropriate resolution (before completing the arrival of a message, for
example) is suppressed because the fractions of seconds that it contains will
never be repeatable.


Philip Hazel
15 February 2006
Commit	Line	Data
75e0e026 PH	1	$Cambridge: exim/doc/doc-txt/test-harness.txt,v 1.1 2006/02/16 10:05:33 ph10 Exp $
	2
	3	EXIM'S BEHAVIOUR CHANGES WHEN RUNNING IN THE TEST HARNESS
	4	---------------------------------------------------------
	5
	6	When Exim is running in its test harness, via the scripts in the exim-testsuite
	7	distribution, its behaviour is altered in a few ways, in order to make the
	8	regression testing work properly. The documentation for the test suite
	9	describes how a copy of the Exim binary is taken and patched in order to get it
	10	to run in the test harness. This document briefly lists the behavioural changes
	11	that result. They come into play when the Boolean variable running_in_test_
	12	harness is true.
	13
	14
	15	Privilege
	16	---------
	17
	18	Exim does not give up its root privilege when called with -C or -D, nor does it
	19	insist on the caller being an admin user when starting a daemon, a queue
	20	runner, or requesting debug output.
	21
	22
	23	Small Pauses
	24	------------
	25
	26	In a number of places, typically when a subprocess has been forked, there are
	27	short pauses of half or one second in one of the processes. This allows the
	28	other process to "go first"; it ensures that debugging or logging output always
	29	appears in the same order.
	30
	31
	32	Daemon
	33	------
	34
	35	The daemon always writes a pid file when running in the test harness.
	36
	37
	38	CRAM-MD5
	39	--------
	40
	41	The cram_md5 authenticator always uses the same challenge string.
	42
	43
	44	Appendfile
	45	----------
	46
	47	After a quota error, the "time since last read" for the file is forced to 10s,
	48	for repeatability.
	49
	50
	51	Memory management
	52	-----------------
	53
	54	Memory management debugging output contains only the store pool and the size
	55	(other information is too variable). New memory is initialized to contain F0 in
	56	all bytes.
	57
	58
	59	Queue running
	60	-------------
	61
	62	There's a facility (-Tqt) for fudging queue times for testing retry logic.
	63
	64
65	Syslog
66	------
67
68	Exim never writes to syslog in the test harness. Attempts to do so are silently
69	ignored. None of the tests actually specify syslog logging for any actual log
70	lines, but there is one that tests the inability to open the main and panic
71	logs, which by default then tries to write to syslog.
72
73
74	SMTP connection timeout
75	-----------------------
76
77	In order to be able to test timeout handling, a "connection refused" error is
78	converted into a timeout if the timeout value is set to 999999s.
79
80
81	Random numbers
82	--------------
83
84	The seed for the pseudo-random number generator is set to a fixed value in the
85	test harness, to ensure repeatability.
86
87
88	Bounce messages
89	---------------
90
91	When Exim is submitting a bounce message to itself, unless the configuration
92	has set queue_only, it uses -odi so that the bounce is delivered before the
93	subprocess returns. This avoids a race that might put log lines in an arbitrary
94	order.
95
96
97	DNS lookups
98	-----------
99
100	The real DNS resolver is never called. Instead, a fake resolver, which runs as
101	a separate program, is used. It is part of the test suite and is documented
102	there. This ensures complete control over the exact results of any DNS lookups.
103
104	An attempt to look up a PTR record for 99.99.99.99 or an IP address for a host
105	whose name ends with .test.again.dns always yields a "try again" error.
106
107	A fake function is called instead of gethostbyname(). It recognizes the name
108	"manyhome.test.ex" and generates a humungous number of IP addresses. It also
109	recognizes an unqualified "localhost" and forces it to the appropriate loopback
110	address (IPv4 or IPv6, as required). IP addresses are treated as literals. For
111	other names, it does a DNS lookup (which of course actually calls the fake
112	resolver) to find the host name.
113
114
115	User names
116	----------
117
118	If unknown_login is set, it forces the login name, thus overriding the actual
119	login for the test suite caller. When this happens, unknown_username provides a
120	user name if it is set; otherwise an empty string is used.
121
122
123	Ident
124	-----
125
126	If -bh is used and both the sending host port and the incoming interface port
127	are supplied, an ident (RFC 1413) call is made for testing purposes.
128
129
130	Debug output
131	------------
132
133	Debugging output from the function that waits for the clock to tick at an
134	appropriate resolution (before completing the arrival of a message, for
135	example) is suppressed because the fractions of seconds that it contains will
136	never be repeatable.
137
138
139	Philip Hazel
140	15 February 2006