Update version number to 4.53.
[exim.git] / doc / doc-txt / NewStuff
dceb978c 1$Cambridge: exim/doc/doc-txt/NewStuff,v 1.56 2005/07/27 18:27:55 fanf2 Exp $
3New Features in Exim
6This file contains descriptions of new features that have been added to Exim,
7but have not yet made it into the main manual (which is most conveniently
8updated when there is a relatively large batch of changes). The doc/ChangeLog
9file contains a listing of all changes, including bug fixes.
11Exim version 4.53
14TK/01 Added the "success_on_redirect" address verification option. When an
15 address generates new addresses during routing, Exim will abort
16 verification with "success" when more than one address has been
17 generated, but continue to verify a single new address. The latter
18 does not happen when the new "success_on_redirect" option is set, like
20 require verify = recipient/success_on_redirect/callout=10s
22 In that case, verification will succeed when a router generates a new
23 address.
495ae4b0 25
26Exim version 4.52
29TF/01 Support for checking Client SMTP Authorization has been added. CSA is a
30 system which allows a site to advertise which machines are and are not
31 permitted to send email. This is done by placing special SRV records in
32 the DNS, which are looked up using the client's HELO domain. At this
33 time CSA is still an Internet-Draft.
35 Client SMTP Authorization checks are performed by the ACL condition
36 verify=csa. This will fail if the client is not authorized. If there is
37 a DNS problem, or if no valid CSA SRV record is found, or if the client
38 is authorized, the condition succeeds. These three cases can be
39 distinguished using the expansion variable $csa_status, which can take
40 one of the values "fail", "defer", "unknown", or "ok". The condition
41 does not itself defer because that would be likely to cause problems
42 for legitimate email.
44 The error messages produced by the CSA code include slightly more
45 detail. If $csa_status is "defer" this may be because of problems
46 looking up the CSA SRV record, or problems looking up the CSA target
47 address record. There are four reasons for $csa_status being "fail":
48 the client's host name is explicitly not authorized; the client's IP
49 address does not match any of the CSA target IP addresses; the client's
50 host name is authorized but it has no valid target IP addresses (e.g.
51 the target's addresses are IPv6 and the client is using IPv4); or the
52 client's host name has no CSA SRV record but a parent domain has
53 asserted that all subdomains must be explicitly authorized.
55 The verify=csa condition can take an argument which is the domain to
56 use for the DNS query. The default is verify=csa/$sender_helo_name.
58 This implementation includes an extension to CSA. If the query domain
59 is an address literal such as [], or if it is a bare IP
60 address, Exim will search for CSA SRV records in the reverse DNS as if
61 the HELO domain was e.g. Therefore it is
62 meaningful to say, for example, verify=csa/$sender_host_address - in
63 fact, this is the check that Exim performs if the client does not say
64 HELO. This extension can be turned off by setting the main
65 configuration option dns_csa_use_reverse = false.
67 If a CSA SRV record is not found for the domain itself, then a search
68 is performed through its parent domains for a record which might be
69 making assertions about subdomains. The maximum depth of this search is
70 limited using the main configuration option dns_csa_search_limit, which
71 takes the value 5 by default. Exim does not look for CSA SRV records in
72 a top level domain, so the default settings handle HELO domains as long
73 as seven (hostname.five.four.three.two.one.com) which encompasses the
74 vast majority of legitimate HELO domains.
76 The dnsdb lookup also has support for CSA. Although dnsdb already
77 supports SRV lookups, this is not sufficient because of the extra
78 parent domain search behaviour of CSA, and (as with PTR lookups)
79 dnsdb also turns IP addresses into lookups in the reverse DNS space.
80 The result of ${lookup dnsdb {csa=$sender_helo_name} } has two
81 space-separated fields: an authorization code and a target host name.
82 The authorization code can be "Y" for yes, "N" for no, "X" for explicit
83 authorization required but absent, or "?" for unknown.
85PH/01 The amount of output produced by the "make" process has been reduced,
86 because the compile lines are often rather long, making it all pretty
87 unreadable. The new style is along the lines of the 2.6 Linux kernel:
88 just a short line for each module that is being compiled or linked.
89 However, it is still possible to get the full output, by calling "make"
90 like this:
92 FULLECHO='' make -e
94 The value of FULLECHO defaults to "@", the flag character that suppresses
95 command reflection in "make". When you ask for the full output, it is
96 given in addition to the the short output.
4df1e33e 98TF/02 There have been two changes concerned with submission mode:
87ba3f5f 99
100 Until now submission mode always left the return path alone, whereas
101 locally-submitted messages from untrusted users have the return path
102 fixed to the user's email address. Submission mode now fixes the return
103 path to the same address as is used to create the Sender: header. If
104 /sender_retain is specified then both the Sender: header and the return
105 path are left alone.
87ba3f5f 106
107 Note that the changes caused by submission mode take effect after the
108 predata ACL. This means that any sender checks performed before the
109 fix-ups will use the untrusted sender address specified by the user, not
110 the trusted sender address specified by submission mode. Although this
111 might be slightly unexpected, it does mean that you can configure ACL
112 checks to spot that a user is trying to spoof another's address, for
113 example.
87ba3f5f 114
115 There is also a new /name= option for submission mode which allows you
116 to specify the user's full name to be included in the Sender: header.
117 For example:
119 accept authenticated = *
120 control = submission/name=${lookup {$authenticated_id} \
121 lsearch {/etc/exim/namelist} }
123 The namelist file contains entries like
125 fanf: Tony Finch
127 And the resulting Sender: header looks like
129 Sender: Tony Finch <fanf@exim.org>
131TF/03 The control = fakereject ACL modifier now has a fakedefer counterpart,
132 which works in exactly the same way except it causes a fake SMTP 450
133 response after the message data instead of a fake SMTP 550 response.
134 You must take care when using fakedefer because it will cause messages
135 to be duplicated when the sender retries. Therefore you should not use
136 fakedefer if the message will be delivered normally.
138TF/04 There is a new ratelimit ACL condition which can be used to measure
139 and control the rate at which clients can send email. This is more
140 powerful than the existing smtp_ratelimit_* options, because those
141 options only control the rate of commands in a single SMTP session,
142 whereas the new ratelimit condition works across all connections
143 (concurrent and sequential) to the same host.
145 The syntax of the ratelimit condition is:
147 ratelimit = <m> / <p> / <options> / <key>
149 If the average client sending rate is less than m messages per time
150 period p then the condition is false, otherwise it is true.
152 The parameter p is the smoothing time constant, in the form of an Exim
153 time interval e.g. 8h for eight hours. A larger time constant means it
154 takes Exim longer to forget a client's past behaviour. The parameter m is
155 the maximum number of messages that a client can send in a fast burst. By
156 increasing both m and p but keeping m/p constant, you can allow a client
157 to send more messages in a burst without changing its overall sending
158 rate limit. Conversely, if m and p are both small then messages must be
159 sent at an even rate.
161 The key is used to look up the data used to calcluate the client's
162 average sending rate. This data is stored in a database maintained by
163 Exim in its spool directory alongside the retry database etc. For
164 example, you can limit the sending rate of each authenticated user,
165 independent of the computer they are sending from, by setting the key
166 to $authenticated_id. The default key is $sender_host_address.
167 Internally, Exim includes the smoothing constant p and the options in
168 the lookup key because they alter the meaning of the stored data.
169 This is not true for the limit m, so you can alter the configured
170 maximum rate and Exim will still remember clients' past behaviour,
171 but if you alter the other ratelimit parameters Exim will effectively
172 forget their past behaviour.
174 Each ratelimit condition can have up to two options. The first option
175 specifies what Exim measures the rate of, and the second specifies how
176 Exim handles excessively fast clients.
178 The per_mail option means that it measures the client's rate of sending
179 messages. This is the default if none of the per_* options is specified.
181 The per_conn option means that it measures the client's connection rate.
183 The per_byte option limits the sender's email bandwidth. Note that it
184 is best to use this option in the DATA ACL; if it is used in an earlier
185 ACL it relies on the SIZE parameter on the MAIL command, which may be
186 inaccurate or completely missing. You can follow the limit m in the
187 configuration with K, M, or G to specify limits in kilobytes,
188 megabytes, or gigabytes respectively.
190 The per_cmd option means that Exim recomputes the rate every time the
191 condition is processed, which can be used to limit the SMTP command rate.
192 The alias per_rcpt is provided for use in the RCPT ACL instead of per_cmd
193 to make it clear that the effect is to limit the rate at which recipients
194 are accepted. Note that in this case the rate limiting engine will see a
195 message with many recipients as a large high-speed burst.
197 If a client's average rate is greater than the maximum, the rate
198 limiting engine can react in two possible ways, depending on the
199 presence of the strict or leaky options. This is independent of the
200 other counter-measures (e.g. rejecting the message) that may be
201 specified by the rest of the ACL. The default mode is leaky, which
202 avoids a sender's over-aggressive retry rate preventing it from getting
203 any email through.
205 The strict option means that the client's recorded rate is always
206 updated. The effect of this is that Exim measures the client's average
207 rate of attempts to send email, which can be much higher than the
208 maximum. If the client is over the limit it will be subjected to
209 counter-measures until it slows down below the maximum rate.
211 The leaky option means that the client's recorded rate is not updated
212 if it is above the limit. The effect of this is that Exim measures the
213 client's average rate of successfully sent email, which cannot be
214 greater than the maximum. If the client is over the limit it will
215 suffer some counter-measures, but it will still be able to send email
216 at the configured maximum rate, whatever the rate of its attempts.
218 As a side-effect, the ratelimit condition will set the expansion
219 variables $sender_rate containing the client's computed rate,
220 $sender_rate_limit containing the configured value of m, and
221 $sender_rate_period containing the configured value of p.
223 Exim's other ACL facilities are used to define what counter-measures
224 are taken when the rate limit is exceeded. This might be anything from
225 logging a warning (e.g. while measuring existing sending rates in order
226 to define our policy), through time delays to slow down fast senders,
227 up to rejecting the message. For example,
229 # Log all senders' rates
230 warn
231 ratelimit = 0 / 1h / strict
232 log_message = \
233 Sender rate $sender_rate > $sender_rate_limit / $sender_rate_period
235 # Slow down fast senders
236 warn
237 ratelimit = 100 / 1h / per_rcpt / strict
238 delay = ${eval: 10 * ($sender_rate - $sender_rate_limit) }
240 # Keep authenticated users under control
241 deny
242 ratelimit = 100 / 1d / strict / $authenticated_id
244 # System-wide rate limit
245 defer
246 message = Sorry, too busy. Try again later.
247 ratelimit = 10 / 1s / $primary_hostname
249 # Restrict incoming rate from each host, with a default rate limit
250 # set using a macro and special cases looked up in a table.
251 defer
252 message = Sender rate $sender_rate exceeds \
253 $sender_rate_limit messages per $sender_rate_period
254 ratelimit = ${lookup {$sender_host_address} \
255 cdb {DB/ratelimits.cdb} \
256 {$value} {RATELIMIT} }
258 Warning: if you have a busy server with a lot of ratelimit tests,
259 especially with the per_rcpt option, you may suffer from a performance
260 bottleneck caused by locking on the ratelimit hints database. Apart from
261 making your ACLs less complicated, you can reduce the problem by using a
262 RAM disk for Exim's hints directory, /var/spool/exim/db/. However this
263 means that Exim will lose its hints data after a reboot (including retry
264 hints, the callout cache, and ratelimit data).
266TK/01 Added an 'spf' lookup type that will return an SPF result for a given
267 email address (the key) and an IP address (the database):
269 ${lookup {tom@duncanthrax.net} spf{}}
271 The lookup will return the same result strings as they can appear in
272 $spf_result (pass,fail,softfail,neutral,none,err_perm,err_temp). The
273 lookup is armored in EXPERIMENTAL_SPF. Currently, only IPv4 addresses
274 are supported.
276 Patch submitted by Chris Webb <chris@arachsys.com>.
278PH/02 There's a new verify callout option, "fullpostmaster", which first acts
279 as "postmaster" and checks the recipient <postmaster@domain>. If that
280 fails, it tries just <postmaster>, without a domain, in accordance with
281 the specification in RFC 2821.
283PH/03 The action of the auto_thaw option has been changed. It no longer applies
284 to frozen bounce messages.
286TK/02 There are two new expansion items to help with the implementation of
287 the BATV "prvs" scheme in an Exim configuration:
290 ${prvs {<ADDRESS>}{<KEY>}{[KEYNUM]}}
292 The "prvs" expansion item takes three arguments: A qualified RFC2821
293 email address, a key and an (optional) key number. All arguments are
294 expanded before being used, so it is easily possible to lookup a key
295 and key number using the address as the lookup key. The key number is
296 optional and defaults to "0". The item will expand to a "prvs"-signed
297 email address, to be typically used with the "return_path" option on
298 a smtp transport. The decision if BATV should be used with a given
299 sender/recipient pair should be done on router level, to avoid having
300 to set "max_rcpt = 1" on the transport.
303 ${prvscheck {<ADDRESS>}{<SECRET>}{<RETURN_STRING>}}
305 The "prvscheck" expansion item takes three arguments. Argument 1 is
306 expanded first. When the expansion does not yield a SYNTACTICALLY
307 valid "prvs"-scheme address, the whole "prvscheck" item expands to
308 the empty string. If <ADDRESS> is a "prvs"-encoded address after
309 expansion, two expansion variables are set up:
311 $prvscheck_address Contains the "prvs"-decoded version of
312 the address from argument 1.
314 $prvscheck_keynum Contains the key number extracted from
315 the "prvs"-address in argument 1.
317 These two variables can be used in the expansion code of argument 2
318 to retrieve the <SECRET>. The VALIDITY of the "prvs"-signed address
319 is then checked. The result is stored in yet another expansion
320 variable:
322 $prvscheck_result Contains the result of a "prvscheck"
323 expansion: Unset (the empty string) for
324 failure, "1" for success.
326 The "prvscheck" expansion expands to the empty string if <ADDRESS>
327 is not a SYNTACTICALLY valid "prvs"-scheme address. Otherwise,
328 argument 3 defines what "prvscheck" expands to: If argument 3
329 is the empty string, "prvscheck" expands to the decoded version
330 of the address (no matter if it is CRYPTOGRAPHICALLY valid or not).
331 If argument 3 expands to a non-empty string, "prvscheck" expands
332 to that string.
335 Usage example
336 -------------
338 Macro:
340 PRVSCHECK_SQL = ${lookup mysql{SELECT secret FROM batv_prvs WHERE \
341 sender='${quote_mysql:$prvscheck_address}'}{$value}}
345 # Bounces: drop unsigned addresses for BATV senders
346 deny message = This address does not send an unsigned reverse path.
347 senders = :
348 recipients = +batv_recipients
350 # Bounces: In case of prvs-signed address, check signature.
351 deny message = Invalid reverse path signature.
352 senders = :
353 condition = ${prvscheck {$local_part@$domain}{PRVSCHECK_SQL}{1}}
354 !condition = $prvscheck_result
356 Top-Level Router:
358 batv_redirect:
359 driver = redirect
360 data = ${prvscheck {$local_part@$domain}{PRVSCHECK_SQL}{}}
362 Transport (referenced by router that makes decision if
363 BATV is applicable):
365 external_smtp_batv:
366 driver = smtp
367 return_path = ${prvs {$return_path} \
368 {${lookup mysql{SELECT \
369 secret FROM batv_prvs WHERE \
370 sender='${quote_mysql:$sender_address}'} \
371 {$value}fail}}}
373PH/04 There are two new options that control the retrying done by the daemon
374 at startup when it cannot immediately bind a socket (typically because
375 the socket is already in use). The default values reproduce what were
376 built-in constants previously: daemon_startup_retries defines the number
377 of retries after the first failure (default 9); daemon_startup_sleep
378 defines the length of time to wait between retries (default 30s).
0cd68797 379
380PH/05 There is now a new ${if condition called "match_ip". It is similar to
381 match_domain, etc. It must be followed by two argument strings. The first
382 (after expansion) must be an IP address or an empty string. The second
383 (after expansion) is a restricted host list that can match only an IP
384 address, not a host name. For example:
386 ${if match_ip{$sender_host_address}{}{...}{...}}
388 The specific types of host list item that are permitted in the list are
389 shown below. Consult the manual section on host lists for further
390 details.
392 . An IP address, optionally with a CIDR mask.
394 . A single asterisk matches any IP address.
396 . An empty item matches only if the IP address is empty. This could be
397 useful for testing for a locally submitted message or one from specific
398 hosts in a single test such as
400 ${if match_ip{$sender_host_address}{:}{...}{...}}
402 where the first item in the list is the empty string.
404 . The item @[] matches any of the local host's interface addresses.
406 . Lookups are assumed to be "net-" style lookups, even if "net-" is not
407 specified. Thus, the following are equivalent:
409 ${if match_ip{$sender_host_address}{lsearch;/some/file}...
410 ${if match_ip{$sender_host_address}{net-lsearch;/some/file}...
412 You do need to specify the "net-" prefix if you want to specify a
413 specific address mask, for example, by using "net24-".
415PH/06 The "+all" debug selector used to set the flags for all possible output;
416 it is something that people tend to use semi-automatically when
417 generating debug output for me or for the list. However, by including
418 "+memory", an awful lot of output that is very rarely of interest was
419 generated. I have changed this so that "+all" no longer includes
420 "+memory". However, "-all" still turns everything off.
e5a9dba6 422
423Version 4.51
426PH/01 The format in which GnuTLS parameters are written to the gnutls-param
427 file in the spool directory has been changed. This change has been made
428 to alleviate problems that some people had with the generation of the
429 parameters by Exim when /dev/random was exhausted. In this situation,
430 Exim would hang until /dev/random acquired some more entropy.
432 The new code exports and imports the DH and RSA parameters in PEM
433 format. This means that the parameters can be generated externally using
434 the certtool command that is part of GnuTLS.
436 To replace the parameters with new ones, instead of deleting the file
437 and letting Exim re-create it, you can generate new parameters using
438 certtool and, when this has been done, replace Exim's cache file by
439 renaming. The relevant commands are something like this:
441 # rm -f new.params
442 # touch new.params
443 # chown exim:exim new.params
444 # chmod 0400 new.params
445 # certtool --generate-privkey --bits 512 >new.params
446 # echo "" >>new.params
447 # certtool --generate-dh-params --bits 1024 >> new.params
448 # mv new.params params
450 If Exim never has to generate the parameters itself, the possibility of
451 stalling is removed.
453PH/02 A new expansion item for dynamically loading and calling a locally-
454 written C function is now provided, if Exim is compiled with
458 set in Local/Makefile. The facility is not included by default (a
459 suitable error is given if you try to use it when it is not there.)
461 If you enable EXPAND_DLFUNC, you should also be aware of the new redirect
462 router option forbid_filter_dlfunc. If you have unprivileged users on
463 your system who are permitted to create filter files, you might want to
464 set forbid_filter_dlfunc=true in the appropriate router, to stop them
465 using ${dlfunc to run code within Exim.
467 You load and call an external function like this:
469 ${dlfunc{/some/file}{function}{arg1}{arg2}...}
471 Once loaded, Exim remembers the dynamically loaded object so that it
472 doesn't reload the same object file in the same Exim process (but of
473 course Exim does start new processes frequently).
475 There may be from zero to eight arguments to the function. When compiling
476 a local function that is to be called in this way, local_scan.h should be
477 included. The Exim variables and functions that are defined by that API
478 are also available for dynamically loaded functions. The function itself
479 must have the following type:
481 int dlfunction(uschar **yield, int argc, uschar *argv[])
483 Where "uschar" is a typedef for "unsigned char" in local_scan.h. The
484 function should return one of the following values:
486 OK Success. The string that is placed in "yield" is put into
487 the expanded string that is being built.
489 FAIL A non-forced expansion failure occurs, with the error
490 message taken from "yield", if it is set.
492 FAIL_FORCED A forced expansion failure occurs, with the error message
493 taken from "yield" if it is set.
495 ERROR Same as FAIL, except that a panic log entry is written.
497 When compiling a function that is to be used in this way with gcc,
498 you need to add -shared to the gcc command. Also, in the Exim build-time
499 configuration, you must add -export-dynamic to EXTRALIBS.
b5aea5e1 500
501TF/01 $received_time is a new expansion variable containing the time and date
502 as a number of seconds since the start of the Unix epoch when the
503 current message was received.
b5aea5e1 504
505PH/03 There is a new value for RADIUS_LIB_TYPE that can be set in
506 Local/Makefile. It is RADIUSCLIENTNEW, and it requests that the new API,
507 in use from radiusclient 0.4.0 onwards, be used. It does not appear to be
508 possible to detect the different versions automatically.
510PH/04 There is a new option called acl_not_smtp_mime that allows you to scan
511 MIME parts in non-SMTP messages. It operates in exactly the same way as
512 acl_smtp_mime
514PH/05 It is now possible to redefine a macro within the configuration file.
515 The macro must have been previously defined within the configuration (or
516 an included file). A definition on the command line using the -D option
517 causes all definitions and redefinitions within the file to be ignored.
518 In other words, -D overrides any values that are set in the file.
519 Redefinition is specified by using '==' instead of '='. For example:
521 MAC1 = initial value
522 ...
523 MAC1 == updated value
525 Redefinition does not alter the order in which the macros are applied to
526 the subsequent lines of the configuration file. It is still the same
527 order in which the macros were originally defined. All that changes is
528 the macro's value. Redefinition makes it possible to accumulate values.
529 For example:
531 MAC1 = initial value
532 ...
533 MAC1 == MAC1 and something added
535 This can be helpful in situations where the configuration file is built
536 from a number of other files.
538PH/06 Macros may now be defined or redefined between router, transport,
539 authenticator, or ACL definitions, as well as in the main part of the
540 configuration. They may not, however, be changed within an individual
541 driver or ACL, or in the local_scan, retry, or rewrite sections of the
542 configuration.
544PH/07 $acl_verify_message is now set immediately after the failure of a
545 verification in an ACL, and so is available in subsequent modifiers. In
546 particular, the message can be preserved by coding like this:
548 warn !verify = sender
549 set acl_m0 = $acl_verify_message
551 Previously, $acl_verify_message was set only while expanding "message"
552 and "log_message" when a very denied access.
554PH/08 The redirect router has two new options, sieve_useraddress and
555 sieve_subaddress. These are passed to a Sieve filter to specify the :user
556 and :subaddress parts of an address. Both options are unset by default.
557 However, when a Sieve filter is run, if sieve_useraddress is unset, the
558 entire original local part (including any prefix or suffix) is used for
559 :user. An unset subaddress is treated as an empty subaddress.
475fe28a 560
561PH/09 Quota values can be followed by G as well as K and M.
563PH/10 $message_linecount is a new variable that contains the total number of
564 lines in the header and body of the message. Compare $body_linecount,
565 which is the count for the body only. During the DATA and
566 content-scanning ACLs, $message_linecount contains the number of lines
567 received. Before delivery happens (that is, before filters, routers, and
568 transports run) the count is increased to include the Received: header
569 line that Exim standardly adds, and also any other header lines that are
570 added by ACLs. The blank line that separates the message header from the
571 body is not counted. Here is an example of the use of this variable in a
574 deny message = Too many lines in message header
575 condition = \
576 ${if <{250}{${eval: $message_linecount - $body_linecount}}}
578 In the MAIL and RCPT ACLs, the value is zero because at that stage the
579 message has not yet been received.
581PH/11 In a ${run expansion, the variable $value (which contains the standard
582 output) is now also usable in the "else" string.
584PH/12 In a pipe transport, although a timeout while waiting for the pipe
585 process to complete was treated as a delivery failure, a timeout while
586 writing the message to the pipe was logged, but erroneously treated as a
587 successful delivery. Such timeouts include transport filter timeouts. For
588 consistency with the overall process timeout, these timeouts are now
589 treated as errors, giving rise to delivery failures by default. However,
590 there is now a new Boolean option for the pipe transport called
591 timeout_defer, which, if set TRUE, converts the failures into defers for
592 both kinds of timeout. A transport filter timeout is now identified in
593 the log output.
7766a4f0 595
f7b63901 596Version 4.50
b9e40c51 599The documentation is up-to-date for the 4.50 release.