Tidy up "make" output along the lines of a 2.6 kernel make (just a short
[exim.git] / doc / doc-txt / Exim4.upgrade
CommitLineData
495ae4b0
PH
1$Cambridge: exim/doc/doc-txt/Exim4.upgrade,v 1.1 2004/10/07 15:04:35 ph10 Exp $
2
3Upgrading Exim from Release 3.33 to 4.xx
4----------------------------------------
5
6Exim 4.00 represents the largest upheaval in Exim's history. There are a lot of
7changes to the way some parts of Exim work, and a lot of incompatible changes
8to the run time configuration file.
9
10This document is in two parts. The first part contains instructions and
11suggestions for how you might go about performing the upgrade. The second part
12is a brief list of all the changes that have taken place. For full details of
13all the new features, please consult the current version of the reference
14manual.
15
16
17HOW TO UPGRADE YOUR EXIM
18------------------------
19
20When you compile Exim 4, a Perl script called convert4r4 is built in the build
21directory. It is not installed by the install script, because it is likely that
22you will run it only once.
23
24This script is provided to assist in updating Exim configuration files. It
25reads an Exim 3 configuration file on the standard input, and writes a modified
26file on the standard output. It also writes comments about what it has done to
27the standard error file. It assumes that the input is a valid Exim 3
28configuration file. A typical call to the conversion script might be
29
30 ./convert4r4 </etc/exim/configure >/etc/exim/configure.new
31
32The output file MUST be checked and tested before trying to use it on a live
33system. The conversion script is just an aid which does a lot of the "grunt
34work". It does not guarantee to produce an Exim 4 configuration that behaves
35exactly the same as the Exim 3 configuration it reads.
36
37Each option change in the new file is preceded by an identifying comment. In
38fact, the conversion script tends to make quite a mess of your configuration,
39and you should expect to go through it afterwards and tidy it up by hand.
40
41Unless you are running a very straightforward configuration, the automatic
42conversion is likely to generate a non-optimal configuration. You should not
43only check it thoroughly, but also run as many tests as you can, to ensure that
44it is working as you expect. In particular, you should test address routing,
45using -bt and -bv, and the policy controls, using -bh. If possible, you should
46also do some live tests (i.e. send and receive some messages) before putting
47Exim 4 into service.
48
49If you have a very complicated configuration, it is possible that convert4r4
50will break it in some situations, which is why thorough testing is strongly
51recommended.
52
53 *********************************
54 ***** You Have Been Warned ******
55 *********************************
56
57
58HOW TO MOVE FROM AN EXIM 3 RELEASE TO AN EXIM 4 RELEASE
59-------------------------------------------------------
60
61One way of upgrading to Exim 4 from a version 3 release is as follows:
62
631. Suppose your run time configuration file is called /usr/exim/configure, and
64 you want to continue with this name after upgrading. The first thing to do
65 is to make another copy of this file called, say, /usr/exim/configure.r3.
66
672. Rebuild your existing Exim to use the copy of the run time configuration
68 file instead of the standard file. Install this version of Exim and HUP your
69 daemon. You can check on the name of the configuration file by running
70
71 exim -bP configure_file
72
73 Ensure that everything is running smoothly. You now have something you can
74 fall back to. IMPORTANT: when you do this re-install, you should also
75 re-install the utilities because four of them (exicyclog, eximon, exinext,
76 and exiwhat) also refer to the configuration file.
77
783. Build the new release, configured to use the standard configuration file.
79
804. Use the convert4r4 utility to upgrade your configuration file for the new
81 release. After running the conversion utility, check the file by hand, and
82 tidy it up.
83
845. Test, test, test! And test some more!
85
866. You can run complete tests, including actual deliveries, from an uninstalled
87 binary, but you have to tell it where it is, so that any re-executions can
88 be done. You can do this by temporarily inserting a setting such as
89
90 exim_path = /source/exim/exim-4.00/build-SunOS5-5.8-sparc/exim
91
92 into the run time configuration. If you want to, you can also insert
93 settings for spool_directory and log_file_path to divert those away from
94 their normal places. Remember to remove these temporary settings when you
95 eventually install the binary for real.
96
977. The new installation script installs the new release as exim-4.00-1, and
98 set a symbolic link called "exim" to point to it. The old version of Exim
99 will be renamed to something like exim-3.33-1.
100
1018. You can now easily change between the new and old releases simply by moving
102 the symbolic link and HUPping your daemon. The format of message files on
103 Exim's spool has _not_ changed, so there should be no problem in changing
104 between releases while there are messages on the queue.
105
1069. HOWEVER: If you do change back and forth between releases, you must also
107 change the utilities exicyclog, eximon, exinext, and exiwhat if you are
108 going to use them. Installing Exim 4 will have left the old versions with a
109 .O suffix. It might be helpful to rename these so that you don't lose them.
110
111
112WHAT HAS NOT CHANGED IN EXIM 4.00
113---------------------------------
114
115The basic overall philosophy, design, and process structure has not changed.
116The format of spool files is the same. The transports have had only minor
117modifications. The command line options remain the same, with a couple of
118additions.
119
120The general run time configuration approach has not changed, but the actual
121details of the configuration file are different.
122
123The Exim monitor has not changed, and there have been only very minor changes
124to other Exim utilities.
125
126
127WHAT HAS CHANGED IN EXIM 4.00
128-----------------------------
129
130The rest of this document lists the very many changes that have taken place.
131I'm going to give only brief details here, because this part of the document is
132intended as a way of alerting you to areas of difference. The reference manual
133describes how the new features work in detail.
134
135
136Named domain, host, address, and local part lists
137-------------------------------------------------
138
139A new feature makes it possible to give names to lists of domains, hosts,
140addresses, and local parts. The syntax used is
141
142 domainlist <name> = <a domain list>
143 hostlist <name> = <a host list>
144 addresslist <name> = <an address list>
145 localpartlist <name> = <a list of local parts>
146
147For example:
148
149 domainlist local_domains = *.my.domain
150 addresslist bad_senders = cdb;/etc/badsenders
151
152These lists are referenced in options by giving the name preceded by a + sign.
153For example, in a router you might have
154
155 domains = +local_domains
156
157At first sight, these lists might seem to be the same as macros, but they are
158not. Macros are just textual substitutions. If you write
159
160 ALIST = host1 : host2
161 auth_advertise_hosts = !ALIST
162
163it probably won't do what you want, because that is exactly the same as
164
165 auth_advertise_hosts = !host1 : host2
166
167Notice that the second host name is not negated. However, if you use a host
168list, and write
169
170 hostlist alist = host1 : host2
171 auth_advertise_hosts = ! +alist
172
173the negation applies to the whole list, and so that is equivalent to
174
175 auth_advertise_hosts = !host1 : !host2
176
177These named lists also have a performance advantage. When Exim is routing an
178address or checking an incoming message, it caches the result of tests on the
179lists. So, if you have a setting such as
180
181 domains = +local_domains
182
183on several of your routers, the actual test is done only for the first one.
184However, this caching works only if there are no expansions within the list
185itself or any sublists that it references. In other words, caching happens only
186if the list is known to be the same each time it is referenced.
187
188By default, there may be up to 16 named lists of each type. This limit can be
189extended by changing a compile-time variable.
190
191The use of domain and host lists is recommended for concepts such as local
192domains, relay domains, and relay hosts. The default configuration is set up
193like this.
194
195
196Processing of domain, host, local part, and address lists
197---------------------------------------------------------
198
199The handling of these lists is now more uniform. Every list is expanded as a
200single string before it is used. (In Exim 3, some options were expanded and
201some weren't, and query-style lookup items were then re-expanded.)
202
203If an expansion is forced to fail, Exim behaves as if the item has not been
204found in the list.
205
206The confusing $key variable has been abolished. When processing a domain list,
207$domain contains the relevant domain and can be used in generating database
208queries. Other appropriate variables are set when processing other kinds of
209list; $sender_host and $sender_host_address for checking incoming hosts and
210$host and $host_address for checking outgoing hosts.
211
212Note that this means that any \ and $ characters in regular expressions must be
213escaped if they appear in one of these lists. The new expansion feature to turn
214off expansion (\N ... \N) which is described below can be helpful here.
215
216IMPORTANT: The details of the processing of address lists has been revised. In
217particular, the handling of the case of an item that is a single-key lookup has
218changed. It no longer looks up the domain on its own before looking up the
219complete address. You need to supply an explicit "*@" before the lookup if you
220want just the domain to be looked up. Please check the manual for full details.
221
222If an item in a host list is the empty string, it matches only when no host is
223defined. If used when checking an incoming message, it matches only when the
224message is arriving by SMTP on the standard input from a local process (using
225-bs). This provides a way of distinguishing between SMTP mail from local
226processes and from remote hosts.
227
228The +allow_unknown and +warn_unknown settings for host lists have been replaced
229by a single item, +include_unknown. By default, failure to find a host name
230when needed causes Exim to behave as if the host does not match the list, but
231if +include_unknown is set, the opposite behaviour happens. Whenever
232+include_unknown is invoked, the incident is logged.
233
234
235Merge of Directors and Routers
236------------------------------
237
238There are no longer any directors in Exim 4. There are just routers. All
239addresses are passed to a single list of routers which typically makes use of
240the "domains" option to choose which way to handle specific groups of domains.
241
242A consequence of this is that the code no longer contains any concept of "local
243domains". However, a typical configuration will probably define a named domain
244list (see above) called local_domains, and use it to control routing something
245like this:
246
247 route_remote:
248 driver = dnslookup
249 domains = ! +local_domains
250 transport = remote_smtp
251 no_more
252
253 system_aliases:
254 ....
255
256The first router does DNS routing for all domains that are not in the named
257list of local domains, and no_more ensures that it is the last router for those
258domains. All other domains fall through to the system_aliases and subsequent
259routers. For a complete configuration example, look at the default
260configuration file in src/configure.default.
261
262
263Router Actions
264--------------
265
266The concept of how the routers work is as follows:
267
268A number of pre-conditions are tested (details below). If any of them fails,
269control is passed to the next router. We say "the router is skipped". Otherwise
270the router is run, and can yield one of several different results:
271
272. accept: The router accepts the address, and either queues it for a transport,
273or generates one or more "child" addresses. Processing the original address
274ceases, unless "unseen" is set on the router, in which case the address is
275passed to the next router. Processing of any child addresses starts with the
276first router by default, or at the router defined by redirect_router if it is
277set. This may be any router in the list.
278
279. decline: The router declines to accept the address because it does not
280recognize it at all. The address is passed to the next router, unless no_more
281is set, in which case the address fails.
282
283. pass: The router recognizes the address, but cannot handle it itself. It
284requests that the address be passed to another router. This overrides no_more.
285By default the address is passed to the next router, but this can be changed by
286setting pass_router. However, in this case (unlike redirect_router) the named
287router must be below the current router (to avoid loops).
288
289. fail: The router determines that the address should fail, and queues it for
290the generation of a bounce message. There is no further processing of the
291original address, unless "unseen" is set.
292
293. defer: The router cannot handle the address at the present time. (For
294example, a database may be offline.) No further processing of the address
295happens in this delivery attempt. It is tried again next time.
296
297. error: There is some error in the router (for example, a syntax error in
298its configuration). The action is as for defer.
299
300
301Router pre-conditions
302---------------------
303
304In Exim 3 there are some strange interactions between the generic options that
305test things before running a director or router and the no_more test that
306happens afterwards.
307
308In Exim 4 it is all more straightforward. If any of the pre-condition tests
309fail, the router is skipped and control passes to the next router. The no_more
310option has an effect only if the router is actually run - that is, if all the
311pre-condition tests succeed. The order in which these tests are run is:
312
313 verify status, expn status, domains, local_parts, check_local_user
314
315If all those match, the debug_print string is output when debugging. Exim then
316goes on to test
317
318 senders, require_files, condition
319
320Note that require_files comes near the end of the list, so you cannot use it to
321check for the existence of a file in which to lookup up a domain, local part,
322or sender. However, as these options are all expanded, you can use the "exists"
323expansion condition to make such tests. The require_files option is intended
324for checking files that the router may be going to use internally, or which are
325needed by a specific transport (e.g. .procmailrc).
326
327In Exim 4, local part prefixes and suffixes are recognized and removed before
328any of the other pre-condition tests are done (in Exim 3 they were removed
329afterwards). Note that this means that the local_parts option now tests the
330local part without its prefix or suffix.
331
332If you want to use local parts that include any affixes in a pre-condition
333test, you can do so by using a "condition" option that uses the variables
334$local_part, $local_part_prefix, and $local_part_suffix as necessary.
335
336
337A New Set of Routers
338--------------------
339
340The two sets of routers and directors of Exim 3 have been replaced by a single
341set of routers for Exim 4. These are as follows:
342
343. accept Always accepts an address. It has no private options.
344
345. dnslookup Routes by DNS lookup (descended from lookuphost).
346
347. ipliteral Routes IP literal addresses (unchanged).
348
349. iplookup Special-purpose lookup router (unchanged).
350
351. manualroute Routes domains from explicit data (descended from domainlist).
352
353. queryprogram Routes addresses by running a program (detail changed).
354
355. redirect Redirects addresses; handles all the functions previously
356 supported by aliasfile, forwardfile, and smartuser without
357 a transport.
358
359
360Saving duplication of effort while routing
361------------------------------------------
362
363Early versions of Exim used to copy the routing of one address for all other
364addresses in the same domain, thereby possibly saving some repeated DNS
365lookups. This feature was removed for release 2.12, after the possibility of
366varying the router actions according to the local part (the local_parts option)
367was added. (In fact, the use of $local_part could have broken it earlier.)
368
369For Exim 4, I have added an option called same_domain_copy_routing to the
370dnslookup and manualroute routers. When one of these routers routes an address
371to a remote transport and this option is set, any other addresses in the
372message that have the same domain are automatically given the same routing, but
373only if the router does not set headers_add or headers_remove, and does not
374`widen' the domain during the routing.
375
376
377Generic Router Options
378----------------------
379
380. The global locally_caseless option is replaced by a generic router option
381 called caseful_local_part. By default, routers handle local parts caselessly.
382
383. check_local_user is now a generic option that is needed to check for a local
384 account. Typically used on redirect (for user's forward files) and on accept
385 (for local deliveries).
386
387. The setting self=local has been removed (since there's no concept of local
388 domains in the code). The same kind of effect can be achieved by using
389 self=reroute or self=pass.
390
391. expn is now a generic router option.
392
393. local_part_prefix and local_part_suffix are now generic router options,
394 replacing prefix and suffix on directors.
395
396. Exim 3 has two logging styles for delivery, depending on whether the domain
397 is a local domain or not. For local domains, the address is given just as the
398 local part - this makes these deliveries easier to spot in the log. In Exim 4
399 there's no concept of local domains, so this functionality cannot be
400 automatic. Instead, there's a generic router option called log_as_local which
401 requests "local-style" logging. This option defaults on for the "accept"
402 router, and off for all the others.
403
404. There's an option called retry_use_local_part which is the default for any
405 router that has check_local_user set, and it applies to routing delays. (The
406 same option for transports applies to transport delays.)
407
408. transport_home_directory and transport_current_directory are new generic
409 options on all routers. They set up default values for home_directory and
410 current_directory on the transport to which they route an address. Any
411 settings in the transport override.
412
413. If transport_home_directory is not set, but check_local_user is set, the
414 user's home directory is used as a default value.
415
416. The special fudge that exists in Exim 3 for handling home_directory settings
417 in forwardfile directors is not needed in Exim 4. It has therefore been
418 removed.
419
420. The new_director option in Exim 3 allows the direction of redirected
421 addresses to start at a given director, instead of the first one. In Exim 4,
422 this option is now called redirect_router. The option is used when a redirect
423 router succeeds, and when a queryprogram router returns a "redirect"
424 response.
425
426. There is a new option called pass_router, which specifies the router to go to
427 when a router "passes" on an address. The named router must follow the
428 current router (to avoid routing loops). Note: if a router declines, control
429 always passes to the next router, unless no_more is set.
430
431. There is a new router option called address_data. This is set to a string
432 which is expanded just before the router is run, that is, after all the
433 pre-tests have succeeded. If the expansion is forced to fail, the router
434 declines. Other expansion failures cause delivery of the address to be
435 deferred.
436
437 When the expansion succeeds, the value is retained with the address, and can
438 be accessed using the variable $address_data. Even if the router declines or
439 passes, the value remains with the address, though it can be changed by
440 another address_data setting on a subsequent router. If a router generates
441 child addresses, the value of $address_data propagates to them.
442
443 The idea of address_data is that you can use it to look up a lot of data for
444 the address once, then then pick out parts of the data later. For example,
445 you could use an LDAP lookup to return a string of the form
446
447 uid=1234 gid=5678 mailbox=/mail/xyz forward=/home/xyz/.forward
448
449 In the transport you could then pick out the mailbox by a setting such as
450
451 file = ${extract{mailbox}{$address_data}}
452
453 This makes the configuration file less messy, and also reduces the number of
454 lookups. (Exim does cache the most recent lookup, but there may be several
455 addresses with different lookups.)
456
457. When a transport is run for several addresses simultaneously, the values of
458 $address_data, $local_part_data, and $domain_data are taken from the first
459 address that the transport handles. However, the order in which multiple
460 addresses are processed is not defined. You therefore need to be careful if
461 you want to use these variables with multiple addresses. The smtp transport
462 is the only one which by default handles multiple addresses.
463
464. When an address is routed by a router with the "unseen" option set, a "clone"
465 address is created, and it starts being routed at the next router. (This is
466 what people expect. In Exim 3 it starts at the top - in simple cases that has
467 the same effect because of the anti-looping rule, but if aliases are involved
468 it sometimes doesn't do what you want.)
469
470. The way that require_files works has been changed. Each item in the list is
471 now separately expanded as the test proceeds. The use of leading ! and +
472 characters is unchanged. However, user and group checking is done differently.
473 Previously, seteuid() was used, but seteuid() is no longer used in Exim (see
474 "Security" below). Instead, Exim now scans along the components of the file
475 path and checks the access for the given uid and gid. It expects "x" access
476 on directories and "r" on the final file. This means that file access control
477 lists (on those operating systems that have them) are ignored.
478
479
480Other Consequences of the Director/Router Merge
481-----------------------------------------------
482
483. The -odqr option is abolished, as there is no inbuilt concept of remote
484 domains.
485
486. The -odqs option is equivalent to queue_smtp_domains = *.
487
488. queue_remote_domains is renamed queue_domains, and applies to any domain.
489
490. The -ql option now suppresses remote delivery; routing always happens.
491
492. The "remote" facility of queue_only_file has been removed.
493
494. The match_directory option for forwardfile and localuser has been entirely
495 abolished. Its function can be achieved using the "condition" option in
496 conjunction with check_local_user.
497
498. When an address is being verified, if it is redirected to a single new
499 address, verification continues with that address. If it is redirected to
500 more than one address, verification ceases with a success result. (In Exim 3,
501 this applied only to aliasing, not to forwarding.)
502
503
504The dnslookup router
505--------------------
506
507This router replaces the lookuphost router of Exim 3. It is much the same,
508except that the "gethostbyname" option has been removed. It now does only DNS
509routing - hence the change of name. Routing using gethostbyname() can be done
510by the manualroute router.
511
512
513The manualroute router
514----------------------
515
516This is the new name for the domainlist router, supposedly to make its function
517clearer and to avoid confusion with the "domainlist" that is used to set up
518named domain lists. Several things have been removed and reorganized.
519
520. The old search mechanism (route_file, route_query, route_queries,
521 search_type) have been removed. Instead there is a new option called
522 route_data, which is an expanded string. It should expand to a single routing
523 entry. If the expansion ends up empty (or is forced to fail), the router
524 declines. The route_list option still exists, for convenient listing of a few
525 inline routes.
526
527. There is no longer any MX processing function in this router. The keywords
528 bydns_mx and bydns_a have been removed, leaving just
529
530 bydns => find IP addresses from address records in the DNS
531 byname => find IP addresses by calling gethostbyname()
532
533 The default lookup type is "byname", and this can be omitted from a route
534 data line. If an IP address is given, both "byname" and "bydns" are ignored
535 (so typically you omit this field).
536
537. The qualify_single and search_parents options have also been removed.
538
539. A transport is always required to be set, unless verify_only is set.
540
541. The host_find_failed option can be set to "decline", to cause the router to
542 decline if it can't find an IP address for a listed host.
543
544. If manualroute routes to a local transport, there is no need to specify
545 byname or bydns in the routing data. Any supplied host list is passed as a
546 string in $host, but $host_address is unset.
547
548
549The queryprogram router
550-----------------------
551
552This router has been re-designed:
553
554. You must now specify a user and group for the program to be run using
555 command_user and (if necessary) command_group. It no longer defaults to
556 "nobody". These options are expanded.
557
558. The command is now split up and each argument expanded separately, as happens
559 for the pipe transport. The command name is also expanded.
560
561. The return value "forcefail" has been renamed "fail", and it causes delivery
562 to fail. (The original usage of "fail" meaning "decline" has finally been
563 removed.)
564
565. The $route_option variable, which queryprogram used to be able to set has
566 been abolished. A facility to set the new $address_data variable replaces it.
567
568. The string returned from queryprogram must now be one of:
569
570 DECLINE
571 FAIL text
572 DEFER text
573 PASS
574 FREEZE text
575 REDIRECT text
576 ACCEPT TRANSPORT=transport HOSTS=host list LOOKUP=byname|bydns DATA=text
577
578The text returned for "redirect" is a list of new addresses. The text for FAIL
579is returned in the SMTP dialogue when the router is run as part of address
580verification. It is also logged. The text for DEFER and FREEZE is just logged.
581
582The data items in the "accept" return can be given in any order, and all are
583optional. If no transport is included in the "accept" return, the router's
584default transport is used. The host list and lookup type are needed only if the
585transport is an smtp transport that does not itself have a host list. The
586default lookup type is "byname". If the "data" field is set, its value is
587placed in the $address_data variable.
588
589
590The redirect router
591-------------------
592
593This router replaces forwardfile, appendfile, and the use of smartuser without
594a transport. It has two mutually exclusive options for specifying the data that
595it uses. If "file" is set, the data is taken from a file. Otherwise "data" must
596be set, and the data is the expanded value of that option.
597
598The data may be an alias list, possibly including special entries such as
599:fail:, or it may be a list of filtering instructions.
600
601If "file" is set, but the file does not exist or is empty, or its contents have
602no effect (entirely comments, or a filter that does nothing), the router
603declines. This also happens if the expansion of "file" is forced to fail. Any
604other expansion failure causes the router to defer.
605
606Ownership of the file is checked if check_local_user is set or if owners is
607set, unless check_owner is explicitly set false.
608
609Likewise, the group is checked if owngroups is set, or if check_local_user is
610set and a modemask not containing 020 is set, unless check_group is explicitly
611set false.
612
613If "data" is set, a forced expansion causes the router to decline. This also
614happens if "data" is an empty string or a string that causes nothing to be
615generated and no action to be taken.
616
617Because "data" is now used for traditional /etc/aliases lookups, an empty alias
618no longer gives an error. It behaves in the same way as :unknown: (which is
619still recognized, but ignored).
620
621. If no_repeat_use is set, the router is skipped if _any_ ancestor of the
622 current address was routed by this router. This pre-test happens before any
623 of the others. (Contrast the default loop avoidance logic, which skips a
624 router if an ancestor with the same local part was routed by the router.)
625
626. If include_directory is set, :include: files are constrained to this
627 directory.
628
629. When an address is redirected to a file or a pipe, $address_file or
630 $address_pipe (as appropriate) is set when expanding the value of
631 file_transport or directory_transport.
632
633. There are new options forbid_filter_readfile and forbid_filter_run to lock
634 out the use of the new ${readfile and ${run expansion items in filters.
635
636. If one_time is set, forbid_pipe, forbid_file, and forbid_filter_reply are
637 forced to be true, and headers_add and headers_remove are forbidden.
638
639
640Generic transport options
641-------------------------
642
643. All remote deliveries are now done in subprocesses running with specified
644 UIDs and GIDs. (Formerly, only remote parallel deliveries were done in
645 subprocesses.) As a result, user and group are now generic options that can
646 be used on all transports. The default for both local and remote transports
647 is to run as the Exim user and group. For remote transports, this should not
648 normally be changed, but if it is, the user or group should be able to access
649 the hints databases, though failure to open a hints database is always
650 ignored.
651
652 If it turns out that a transport user is in the never_users list, Exim now
653 defers delivery and writes to the panic log. (Previously it just ran the
654 delivery as "nobody".) Because subprocesses (usually running as "exim")
655 are now always used for remote deliveries, you should *not* include "exim" in
656 the never_users list.
657
658. initgroups is now also a generic transport option.
659
660. home_directory and current_directory are generic options on all transports,
661 though some transports (e.g. smtp) make no use of them. If they are unset,
662 values supplied by the router are used.
663
664. The message_size_limit option is now expanded, which makes it possible to
665 have different limits for different hosts, for example.
666
667
668Multiple (batch) deliveries in the appendfile, lmtp, and pipe transports
669------------------------------------------------------------------------
670
671The options controlling batch deliveries, including BSMTP, were a mess, and
672have been reworked.
673
674. The batch option has been removed from all three transports, and the bsmtp
675 and bsmtp_helo options have been removed from appendfile and pipe.
676
677. The batch_max option defaults to 1 in all three transports.
678
679. A new option called use_bsmtp has been added to appendfile and pipe. When
680 set, the message is delivered in BSMTP format. If you want to have a HELO
681 line at the start of the message, you can configure this by making use of the
682 message_prefix option. You must include the terminating newline.
683
684. A new option called batch_id has been added to all three transports.
685
686Batching is now achieved by setting batch_max to a value greater than 1. This
687is recommended for lmtp. When multiple addresses are routed to the same
688transport that has a batch_max value greater than one, the addresses are
689delivered in a batch, subject to certain conditions:
690
691. If any of the transport's options contain a reference to "$local_part", no
692 batching is possible.
693
694. If any of the transport's options contain a reference to "$domain", only
695 addresses with the same domain are batched.
696
697. If batch_id is set, it is expanded for each address, and only those addresses
698 with the same expanded value are batched.
699
700. Batched addresses must also have the same errors address (where to send
701 delivery errors), the same header additions and removals, the same user and
702 group for the transport, and if a host list is present, the first host must
703 be the same.
704
705
706The appendfile transport
707------------------------
708
709. The prefix and suffix options have been renamed message_prefix and
710 message_suffix to avoid confusion with address affixes. The default values,
711 which are suitable for mbox deliveries, now apply only if "file" is set and
712 use_bsmtp is not set. Otherwise, the default values for these options are
713 unset. They can, of course, always be overridden.
714
715. If "directory" is set (which means that "file" is not set), the check_string
716 and escape_string options now default unset.
717
718. The require_lockfile options has been abolished. If use_lockfile is set, a
719 lock file is always required.
720
721. The quota_filecount option is now expanded.
722
723. The create_file option now also applies when delivering into an individual
724 file in a given directory, as well as when appending to a single file. In the
725 case of maildir delivery, the restriction applies to the top directory of the
726 maildir folder.
727
728. There's a new option called directory_file which is expanded to form the
729 final leaf name of files when "directory" is set, but neither maildir nor
730 mailstore is set. The default is "q${base62:$tod_epoch}-$inode", which
731 reproduces the old fixed value. The variable $inode is available only when
732 expanding this new option.
733
734
735The pipe transport
736------------------
737
738. The prefix and suffix options have been renamed message_prefix and
739 message_suffix to avoid confusion with address affixes. The default values
740 that are suitable for vacation deliveries now apply only if use_bsmtp is not
741 set. Otherwise the default values for these options are unset. They can, of
742 course, always be overridden.
743
744
745The smtp transport
746------------------
747
748. The badly-named batch_max option is now called connection_max_messages.
749
750. If hosts_randomize is set, it now affects host lists that come from a router
751 as well as the contents of the "hosts" option, but only if the hosts were not
752 obtained from MX records. Typically, such lists come from the manualroute
753 router. This change means that the router can provide the same host list for
754 multiple addresses - causing them all to be sent to the transport at once.
755 Randomizing is then done each time the transport is called. (If you set
756 hosts_randomize on the router, the randomizing happens for each address.)
757
758. The way that smtp operates when there are multiple addresses to be sent to
759 the same host is now different. Previously, the transport was called many
760 times, with a maximum of max_rcpt addresses per call. Each call made a new
761 connection to the host. When remote_max_parallel = 1, all the addresses are
762 now passed to the transport at once. It makes a single TCP/IP call, but may
763 send multiple copies of the message, each with no more than max_rcpt
764 recipients.
765
766 When remote_max_parallel is greater than 1, a heuristic is used. The number
767 of addresses passed to a single call of the transport is limited to
768
769 (the total number of recipients) / (the value of remote_max_parallel)
770
771 so, for example, if there are 100 recipients and remote_max_parallel is 2, no
772 more than 50 are passed in one call, even if max_rcpt is 100. (The idea is
773 that the other 50 will be passed to another call running in parallel.)
774
775 There is an option of the smtp transport called connection_max_messages
776 which limits the number of messages, or copies of a message, that Exim sends
777 down a single TCP/IP connection. This applies both to this mechanism for
778 multiple copies of a single message, and the re-use of a TCP/IP connection
779 for sending other messages destined for the same host, after a delivery
780 delay. The default value is 500.
781
782. The "interface" option is now expanded. If the result is a forced failure or
783 an empty string, it is ignored. Otherwise, the result must be a list of IP
784 addresses. The first one of the correct type (IPv4 or IPv6) for the outgoing
785 connection is used. If there isn't one of the correct type, the option is
786 ignored.
787
788. At the start of running the transport, the value of $host is taken from the
789 first host in a multi-host list. However, just before the transport connects
790 to a host, the value is changed to refer to that particular host. (This
791 applies to $host_address as well.) This means that options such as helo_data
792 and the tls_options can be made host-specific.
793
794. The tls_verify_ciphers option has been renamed tls_require_ciphers, in order
795 to leave the word "verify" as something that refers to the verification of
796 certificates.
797
798. The resolution of hosts and fallback_hosts used to look up MX records. This
799 was some kind of ancient silliness that I recently noticed. These are
800 definitely hosts, not mail domains. Exim 4 just looks up address records.
801 As a consequence of this, the mx_domains option of the smtp transport is
802 removed.
803
804. The authenticate_hosts option has been renamed as hosts_try_auth. A new
805 option called hosts_require_auth has been added; if authentication fails for
806 one of these hosts, Exim does _not_ try to send unauthenticated. It defers
807 instead. The deferal error is detectable in the retry rules, so this can be
808 turned into a hard failure if required.
809
810
811The System Filter
812-----------------
813
814. The system filter options that were called message_filter_xxx have all been
815 renamed as system_filter_xxx.
816
817. The value of system_filter is expanded.
818
819. message_filter_directory_transport and message_filter_file_transport are now
820 both expanded before use. If the filter set up any file or pipe deliveries,
821 $address_file and $address_pipe are set as appropriate while doing the
822 expansions.
823
824. message_filter_directory2_transport has been removed. The effect of using
825 different directory-style transports can be achieved by specifying a suitable
826 expansion string to system_filter_directory_transport.
827
828. When a system filter added recipients to a message, Exim 3 added an
829 X-Envelope-To: header, listing the real recipients (up to 100). This has been
830 abolished because you can do this kind of thing using "headers_add" nowadays.
831
832. The "fail" command has been extended to allow for two different messages, one
833 for Exim's log and the other to be returned to the sender. The syntax is
834
835 fail "<<log message>>user message"
836
837 That is, if the first two characters of the message are "<<" the following
838 text, up to ">>", is written to the log, and the remainder is returned to the
839 user. If there is no log message, the user message is logged. The motivation
840 for this feature was to reduce the amount of text logged, while being able to
841 send quite long (maybe even multi-line) messages back to the sender.
842
843
844Changes to Lookups
845------------------
846
847. Oracle support is available. It works like the mysql and pgsql support,
848 except that there is no "database name" involved, and the "host name" field
849 is used for what is called "service name" in Oracle. This often looks like a
850 host name. Also, semicolons are not used at the end of an SQL query for
851 Oracle.
852
853. There's a new single-key lookup type called dsearch. It searches a directory
854 for a file whose name matches the key. The result of a successful search is
855 the key. One possible use of this could be for recognizing virtual domains.
856 If each domain is represented by a file whose name is the domain name, you
857 needn't make a separate list of the domains. You could test for them in an
858 ACL (see below), for example, by a line like this
859
860 accept domains = dsearch;/etc/virtual/domains
861
862. The format of LDAP output has been changed for cases where multiple
863 attributes are requested. The data for each attribute is now always quoted.
864 Within the quotes, the quote character, backslash, and newline are escaped
865 with backslashes and commas are used to separate multiple values for the
866 attribute. Thus, the string in quotes takes the same form as the output when
867 a single attribute is requested. If multiple entries are found, their data is
868 still separated by a newline.
869
870. There's a new expansion condition called ldapauth which exists so that the
871 LDAP authentication mechanism can be used for user authentication. It is
872 described under "string expansion" below.
873
874. Exim now supports ldaps:// URLs as well as ldap:// URLs. The former do LDAP
875 over TLS (i.e. encrypted) connections.
876
877. There is now support for the "whoson" mechanism for doing "POP-before-SMTP"
878 authentication. This is provided by new query-style lookup type called
879 "whoson", with queries that consist of IP addresses. For example, in an ACL
880 you can write
881
882 require condition = ${lookup whoson {$sender_host_address}{yes}{no}}
883
884
885Special items in domain and host lists
886--------------------------------------
887
888. In a domain list, the special item @ matches the primary host name, and the
889 special item @[] matches any local interface address enclosed in square
890 brackets (as in domain literal email addresses). The special item @mx_any
891 matches any domain that has an MX record pointing to the local host. The
892 special items @mx_primary and @mx_secondary are similar, except that the
893 first matches only when the primary MX is to the local host, and the second
894 only when the primary MX is not the local host, but a secondary MX is.
895
896. In a host list, the special item @ matches the primary host name, and the
897 special item @[] matches any local interface address (not in brackets).
898
899
900Access Control Lists (ACLs)
901---------------------------
902
903All the policy control options for incoming messages have been replaced by
904Access Control Lists (ACLs). These give more flexibility to the sysadmin, and
905allow the order of testing to be specified. For example, using an ACL, it is
906possible to specify "accept if authenticated, even if from an RBL host, but
907otherwise deny if from an RBL host", which is not possible in Exim 3.
908
909ACLs are defined in a new part of the configuration file, and given names.
910Which ones to run are controlled by a new set of options that are placed in the
911main part of the configuration.
912
913 acl_smtp_auth specifies the ACL to run when AUTH is received
914 acl_smtp_data specifies the ACL to run after a message has been received
915 acl_smtp_etrn specifies the ACL to run when ETRN is received
916 acl_smtp_expn specifies the ACL to run when EXPN is received
917 acl_smtp_rcpt specifies the ACL to run when RCPT is received
918 acl_smtp_vrfy specifies the ACL to run when VRFY is received
919
920The default actions vary. If acl_smtp_auth is not defined, AUTH is always
921accepted (and an attempt is made to authenticate the session). If acl_smtp_data
922is not defined, no checks are done after a message has been received, and it is
923always accepted at that point.
924
925However, if any of the others are not defined, the relevant SMTP command is
926rejected. In particular, this means that acl_smtp_rcpt must be defined in order
927to receive any messages over an SMTP connection. The default configuration file
928contains a suitable default for this.
929
930ACLs can be provided in line, or in files, or looked up from databases. One ACL
931can call another in a subroutine-like manner. String expansion is used, and
932which ACL to run can be varied according to sender host or any other criterion
933that a string expansion can test.
934
935This is not the place to give a full specification of ACLs, but here is a
936typical example for checking RCPT commands, taken from the default
937configuration. The tests are performed in order.
938
939acl_check_rcpt:
940 # Accept if source is local SMTP (i.e. not over TCP/IP - undefined host)
941 accept hosts = :
942
943 # Deny if the local part contains @ or % or /
944 deny local_parts = ^.*[@%/]
945
946 # Accept mail to postmaster in any local domain, regardless of the source,
947 # and without verifying the sender.
948 accept domains = +local_domains
949 local_parts = postmaster
950
951 # Deny unless the sender address can be verified.
952 require verify = sender
953
954 # Accept if the address is in a local domain, but only if the recipient can
955 # be verified. Otherwise deny. The "endpass" line is the border between
956 # passing on to the next ACL statement (if tests above it fail) or denying
957 # access (if tests below it fail).
958 accept domains = +local_domains
959 endpass
960 message = unknown user
961 verify = recipient
962
963 # We get here only for non-local domains. Accept if the message arrived over
964 # an authenticated connection, from any host. These messages are usually from
965 # MUAs, so recipient verification is omitted.
966 accept authenticated = *
967
968 # Reaching the end of the ACL causes a "deny", but we might as well give
969 # an explicit message.
970 deny message = relay not permitted
971
972The following options have been abolished as a consequence of the introduction
973of ACLs:
974
975auth_hosts, auth_over_tls_hosts, headers_checks_fail, headers_check_syntax,
976headers_sender_verify, headers_sender_verify_errmsg, host_accept_relay,
977host_auth_accept_relay, host_reject_recipients, prohibition_message,
978rbl_domains, rbl_hosts, rbl_log_headers, rbl_log_rcpt_count,
979rbl_reject_recipients, rbl_warn_header, receiver_try_verify, receiver_verify,
980receiver_verify_addresses, receiver_verify_hosts, receiver_verify_senders,
981recipients_reject_except, recipients_reject_except_senders, relay_domains,
982relay_domains_include_local_mx, relay_match_host_or_sender,
983sender_address_relay, sender_address_relay_hosts, sender_reject,
984sender_reject_recipients, sender_try_verify, sender_verify,
985sender_verify_batch, sender_verify_hosts, sender_verify_fixup,
986sender_verify_hosts_callback, sender_verify_callback_domains,
987sender_verify_callback_timeout, sender_verify_max_retry_rate,
988sender_verify_reject, smtp_etrn_hosts, smtp_expn_hosts. smtp_verify, tls_hosts.
989
990The variable $prohibition_reason has been abolished.
991
992The host_reject option has been retained, but with its name changed to
993host_reject_connection, to emphasize that it causes a rejection at connection
994time. I've left it available just in case it is needed - but its use is not
995recommended in normal circumstances.
996
997
998Other Incoming SMTP Session Controls
999------------------------------------
1000
1001. The option smtp_accept_max_per_connection (default 1000) limits the number of
1002 messages accepted over a single SMTP connection. This is a safety catch in
1003 case some sender goes mad (incidents of this kind have been seen). After the
1004 limit is reached, a 421 response is given to MAIL commands.
1005
1006. Some sites find it helpful to be able to limit the rate at which certain
1007 hosts can send them messages, and the rate at which an individual message can
1008 specify recipients. There are now options for controlling these two different
1009 rates.
1010
1011 Rate limiting applies only to those hosts that match smtp_ratelimit_hosts,
1012 whose value is a host list. When a host matches, one or both of the options
1013 smtp_ratelimit_mail and smtp_ratelimit_rcpt may be set. They apply to the
1014 rate of acceptance of MAIL and RCPT commands in a single SMTP session,
1015 respectively.
1016
1017 The value of each option is a set of four comma-separated values:
1018
1019 1. A threshold, before which there is no rate limiting.
1020 2. An initial time delay. Unlike other times in Exim, fractions are allowed
1021 here.
1022 3. A factor by which to increase the delay each time.
1023 4. A maximum value for the delay.
1024
1025 For example, these settings have been used successfully at the site which
1026 first suggested this feature, for controlling mail from their customers:
1027
1028 smtp_ratelimit_mail = 2, 0.5s, 1.05, 4m
1029 smtp_ratelimit_rcpt = 4, 0.25s, 1.015, 4m
1030
1031. The default value for smtp_connect_backlog has been increased to 20.
1032
1033. The SMTP protocol specification requires the client to wait for a response
1034 from the server at certain points in the dialogue. (Without PIPELINING these
1035 are after every command; with PIPELINING they are fewer, but still exist.)
1036 Some spamming sites send out a complete set of SMTP commands without waiting
1037 for any response. Exim 4 protects against this by rejecting messages if the
1038 client has sent further input when it should not have. The error response
1039 "554 SMTP synchronization error" is sent, and the connection is dropped.
1040
1041 This check is controlled by smtp_enforce_sync, which is true by default.
1042
1043. helo_strict_syntax has been abolished. The default is now to enforce strict
1044 domain syntax for HELO/EHLO arguments. You can use helo_accept_junk_hosts if
1045 you want to avoid this.
1046
1047. There's a new option called helo_lookup_domains. If the domain given in a
1048 HELO or EHLO command matches this list, a reverse lookup is done in order to
1049 establish the host's true name. The default setting is
1050
1051 helo_lookup_domains = @ : @[]
1052
1053 That is, a lookup is forced if the client host gives the server's name or
1054 [one of its IP addresses] in HELO or EHLO. (In Exim 3 this happened
1055 automatically and was not configurable.)
1056
1057. The value of the global message_size_limit option is now expanded. For
1058 locally submitted messages this happens at the start of message reception.
1059 For messages from remote hosts, the expansion is done just after the host
1060 connects, so that the value can depend on the host.
1061
1062
1063Handling of Resent- Fields
1064--------------------------
1065
1066RFC 2822 makes it clear that Resent- fields are purely informational. Exim used
1067to make use of Resent-Reply-To: which does not actually exist, and it also used
1068to use the last set of resent- fields for all the address fields it recognized.
1069
1070In Exim 4, resent- headers are dealt with as follows:
1071
1072. A Resent-From: header that just contains the login id as the address is
1073 automatically rewritten in the same way as From: is (using qualify domain,
1074 and user name from the passwd data).
1075
1076. If there's a rewrite rule for a header, it is also applied to resent- headers
1077 of the same type. For example, a rule that rewrites From: headers also
1078 rewrites Resent-From: headers.
1079
1080. For local messages, if Sender: is being removed on input, Resent-Sender: is
1081 also removed.
1082
1083. If there are any resent- headers but no Resent-Date: or Resent-From: they are
1084 added.
1085
1086. The logic for adding Sender: is now duplicated for Resent-Sender.
1087
1088. If there's no Resent-Message-Id: one is created, and it is the
1089 Resent-Message-Id: which is included in the log line.
1090
1091
1092Authentication
1093--------------
1094
1095. The auth_hosts option has been abolished; this functionality is now
1096 controlled by ACLs.
1097
1098. The auth_always_advertise option has been abolished because it depended on
1099 auth_hosts and and host_auth_accept_relay, both of which are no more. In its
1100 place there is a new option called auth_advertise_hosts, whose default value
1101 is *, meaning "advertise AUTH to all".
1102
1103. The value of server_setid is now used when logging failed authentication
1104 attempts.
1105
1106. The -oMaa option allows trusted users to set the value of
1107 $sender_host_authenticated (the authenticator name). This is normally used in
1108 conjunction with -oMa.
1109
1110
1111Encryption
1112----------
1113
1114. Because tls_hosts is no more, tls_advertise_hosts is now the only means of
1115 controlling the advertisement of STARTTLS (previously, tls_hosts overrode).
1116
1117. The global option tls_verify_ciphers has been abolished. There are now
1118 facilities for checking which cipher is in use in ACLs.
1119
1120. There's a new option called tls_try_verify_hosts. Like tls_verify_hosts, this
1121 causes the server to request a certificate from a client, and it verifies the
1122 certificate that it receives. However, unlike tls_verify_hosts, Exim
1123 continues with the SMTP connection (encrypted) if a client certificate is not
1124 received, or if the certificate does not verify. This state can be detected
1125 in an ACL, which makes it possible to implement policies such as "accept for
1126 relay only if a verified certificate has been received but accept for local
1127 delivery if encrypted, even without a verified certificate".
1128
1129 A match in tls_verify_hosts overrides tls_try_verify_hosts.
1130
1131
1132The Daemon
1133----------
1134
1135. local_interfaces can now specify a port number with each address, thus
1136 allowing a single Exim daemon to listen on multiple ports. The format of each
1137 address is either [aaaa]:ppp or aaaa.ppp where aaaa is an IP address and ppp
1138 is a port number. For example:
1139
1140 local_interfaces = 192.168.3.4.25 : 192.168.3.4.26
1141
1142 If an address is listed without a port, the setting of daemon_smtp_port, or
1143 the value of the -oX option, is the default.
1144
1145. The -oX option can now override local_interfaces. That is, it can supply IP
1146 addresses as well as just a port. It is interpreted in this way if its value
1147 contains any of the characters . : or []. For example:
1148
1149 exim -bd -oX 10.9.8.7:10.11.12.13.2525
1150
1151 The format of the string is identical to the format recognized by the
1152 local_interfaces option.
1153
1154. The way the daemon wrote PID files was overly complicated and messy. It no
1155 longer tries to be clever. A PID file is written if, and only if, -bd is used
1156 and -oX is _not_ used. In other words, only if the daemon is started with its
1157 standard options. There is only one PID file. If pid_file_path is unset, it
1158 is exim-daemon.pid in Exim's spool directory. Otherwise the value of
1159 pid_file_path is used. For backwards compatibility, "%s" in this value is
1160 replaced by an empty string.
1161
1162
1163Logging
1164-------
1165
1166The log_level option and all the various independent logging control options
1167have been abolished. In their place there is a single option called
1168log_selector. It takes a string argument composed of names preceded by + or -
1169characters. These turn on or off the logging of different things. For example:
1170
1171 log_selector = +arguments -retry_defer
1172
1173The optional logging items (defaults marked *) are:
1174
1175 address_rewrite address rewriting
1176 all_parents all parents in => lines
1177 arguments exim arguments
1178 *connection_reject connection rejections
1179 *delay_delivery immediate delivery delayed (message queued)
1180 delivery_size add S=nnn to delivery lines
1181 *dnslist_defer defers of DNS list (aka RBL) lookups
1182 incoming_interface incoming interface on <= lines
1183 incoming_port incoming port on <= lines
1184 *lost_incoming_connection as it says (includes timeouts)
1185 *queue_run start and end queue runs
1186 received_sender sender on <= lines
1187 received_recipients recipients on <= lines
1188 *retry_defer "retry time not reached"
1189 sender_on_delivery add sender to => lines
1190 *size_reject rejection because too big
1191 *skip_delivery "message is frozen"
1192 smtp_confirmation SMTP confirmation on <= lines
1193 smtp_connection SMTP connections
1194 smtp_protocol_error SMTP protocol errors
1195 smtp_syntax_error SMTP syntax errors
1196 subject contents of Subject: on <= lines
1197 *tls_cipher TLS cipher on <= lines
1198 tls_peerdn TLS peer DN on <= lines
1199
1200 all all of the above
1201
1202"retry time not reached" is always omitted from individual message logs after
1203the first delivery attempt.
1204
1205The log line "error message sent to" has been abolished, because the R= item on
1206the incoming message line gives the relationship between the original message
1207and the bounce.
1208
1209The logging options that have been abolished are: log_all_parents,
1210log_arguments, log_incoming_port, log_interface, log_ip_options,
1211log_level, log_queue_run_level, log_received_sender, log_received_rceipients,
1212log_rewrites, log_sender_on_delivery, log_smtp_confirmation,
1213log_smtp_connections, log_smtp_syntax_errors, log_subject, tls_log_cipher,
1214tls_log_peerdn.
1215
1216
1217Debugging
1218---------
1219
1220The debug_level option has been removed. The -dm option has been removed. The
1221-df option has also be removed, along with its related build-time option
1222STDERR_FILE. (To debug inetd usage, an auxiliary script should be used.)
1223
1224The -d option has been reworked. It no longer takes a debug level number
1225argument, but instead takes a list of debugging names, each preceded by + or -
1226to turn on or off individual sets of debugging messages.
1227
1228. The -v option now shows just the SMTP dialog and any log lines.
1229
1230. -d with no argument gives a lot of standard debugging data. This is in effect
1231 the equivalent of the old -d9, the thing you ask people to set for an initial
1232 debugging test.
1233
1234. -d+x adds debugging option x to the default set
1235 -d-x removes debugging option x from the default set
1236 -d-all+x leaves only debugging option x
1237
1238The available debugging names are:
1239
1240 acl ACL interpretation
1241 auth authenticators
1242 deliver general delivery logic
1243 dns DNS lookups (see also resolver)
1244 dnsbl DNS black list (aka RBL) code
1245 exec arguments for execv() calls
1246 filter filter handling
1247 hints_lookup hints data lookups
1248 host_lookup all types of name->IP address handling
1249 ident ident lookup
1250 interface lists of local interfaces
1251 lists matching things in lists
1252 load system load checks
1253 lookup general lookup code and all lookups
1254 memory memory handling (replaces the old -dm)
1255 process_info setting info for the process log
1256 queue_run queue runs
1257 receive general message reception logic
1258 resolver turn on the DNS resolver's debugging output; goes to stdout
1259 retry retry handling
1260 rewrite rewriting
1261 route address routing
1262 tls TLS logic
1263 transport transports
1264 uid changes of uid/gid and looking up uid/gid
1265 verify address verification logic
1266
1267 all all of the above, and also -v
1268
1269The default (-d with no argument) includes all of the above, plus -v, with the
1270exception of filter, interface, load, memory, and resolver. Some debugging
1271output always happens unconditionally whenever any debugging is selected. This
1272includes some initial output and every log line.
1273
1274-d without any value was previously allowed for non-admin users because it used
1275to be synonymous with -v. In Exim 4, non-admin users may use -v, but not -d.
1276
1277If the debug_print option is set in any driver, it produces output whenever any
1278debugging is selected, or if -v is used.
1279
1280
1281Local Scan Function
1282-------------------
1283
1284For customized message scanning, you can now supply a C function that is linked
1285into the Exim binary. The function is called local_scan(), and it is called
1286when Exim has received a message, but has not yet sent a final
1287acknowledgement to the sender. This applies to all messages, whether local or
1288remote, SMTP or not.
1289
1290From within your function you can inspect the message, change the recipients,
1291add or remove headers, and tell Exim whether to accept or reject the message.
1292
1293The manual contains the specification of the API for this function.
1294
1295
1296String Expansion
1297----------------
1298
1299. The lookup feature that allowed for subkeys using the syntax
1300
1301 ${lookup {key:subkey} type {data...
1302
1303 has been abolished (a) because the effect can be achieved using ${extract,
1304 and (b) because in non-lsearch lookups, a colon can be a valid character in a
1305 key.
1306
1307. When a string key is used in a ${extract expansion item, it is now handled
1308 case-insensitively.
1309
1310. A new expansion variable called $tod_epoch gives the time as a single decimal
1311 number representing the number of seconds from the start of the Unix epoch.
1312
1313. There's a new expansion operator that can turn numbers into base 62, for
1314 example, ${base62:$tod_epoch}.
1315
1316. ${extract{number} now recognizes a negative number as a request to count
1317 fields from the right.
1318
1319. There's a new expansion feature for reading files:
1320
1321 ${readfile{/some/file}{eolstring}}
1322
1323 The contents of the file replace the item. If {eolstring} is present (it's
1324 optional) any newlines in the file are replaced by that string.
1325
1326. There's a new expansion feature for running commands:
1327
1328 ${run{comand args}{yes}{no}}
1329
1330 Like all the other conditional items, the {yes} and {no} strings are
1331 optional. Omitting both is equivalent to {$value}. The standard output of the
1332 command is put into $value if the command succeeds (returns a zero code). The
1333 value of the code itself is put into $runrc, and this remains set afterwards,
1334 so in a filter file you can do things like
1335
1336 if "${run{x y z}{}}$runrc" is 1 then ...
1337 elsif $runrc is 2 then ...
1338
1339 As in other command executions from Exim, a shell is not used by default.
1340 If you want a shell, you must explicitly code it.
1341
1342. The redirect router has options for forbidding ${readfile and ${run in
1343 filters.
1344
1345. A feature is provided to suppress expansion of part of a string. Any
1346 characters between two occurrences of \N are copied to the output string
1347 verbatim. This is particularly useful for protecting regular expressions from
1348 unwanted expansion effects. For example:
1349
1350 queue_smtp_domains = ! \N^ten-\d+\.testing\.com$\N
1351
1352 Without \N the \ and $ characters in the regex would have to be escaped.
1353
1354. Radius authentication is supported in a similar way to PAM. You must set
1355 RADIUS_CONFIG_FILE in Local/Makefile to specify the location of the Radius
1356 client configuration file. Then you can use expansions such as
1357
1358 server_condition = ${if radius{arguments}{yes}{no}}
1359
1360. User authentication can now also be done by attempting to bind to an LDAP
1361 server. The syntax is again similar to PAM and Radius.
1362
1363 server_condition = ${if ldapauth{ldap query}{yes}{no}}
1364
1365 A user and password are required to be supplied with the query. No actual
1366 data is looked up; Exim just does a bind to the LDAP server and sets the
1367 condition according to the result. Here's an example of an SMTP
1368 authenticator:
1369
1370 login:
1371 driver = plaintext
1372 public_name = LOGIN
1373 server_prompts = "Username:: : Password::"
1374 server_condition = ${if ldapauth \
1375 {user="uid=${quote_ldap:$1},ou=people,o=example.org" pass="$2" \
1376 ldap://ldap.example.org/}{yes}{no}}
1377 server_set_id = uid=$1,ou=people,o=example.org
1378
1379
1380
1381Security
1382--------
1383
1384Exim 3 could be run in a variety of ways as far as security was concerned. This
1385has all been simplified in Exim 4. The security-conscious might like to know
1386that it no longer makes any use of the seteuid() function.
1387
1388. A UID and GID are required to be specified when Exim is compiled. They can be
1389 now specified by name as well as by number, so the relevant options are now
1390 called EXIM_USER and EXIM_GROUP. If you really feel you have to run Exim as
1391 root, you can specify root here, but it is not recommended.
1392
1393. The "security" option has been abolished. Exim always releases its root
1394 privilege when it can. In a conventional configuration, that means when it is
1395 receiving a message, when it is delivering a message, when it is running a
1396 queryprogram router, and when it is obeying users' filter files (and system
1397 filters if it has been given a user for that purpose).
1398
1399. One important change is that Exim 4 runs as root while routing addresses for
1400 delivery. Exim 3 used seteuid() to give up privilege temporarily while
1401 routing. Apart from the unliked use of seteuid(), this sometimes gave rise to
1402 permissions problems on configuration files.
1403
1404. However, Exim still runs as the Exim user while receiving messages, and
1405 therefore while using the routing logic for verifying at SMTP time.
1406
1407. There is a new option called deliver_drop_privilege. If this is set, Exim
1408 gives up its privilege right at the start of a delivery process, and runs the
1409 entire delivery as the Exim user. This is the same action that used to be
1410 requested by setting security=unprivileged.
1411
1412
1413Hints Databases
1414---------------
1415
1416. A single "misc" hints database is now used for ETRN and host serialization.
1417 There have been appropriate consequential changes to the utilities for
1418 managing the hints.
1419
1420. The exim_tidydb -f option has been abolished. A full tidy is now always done
1421 (it hasn't proved to be very expensive).
1422
1423
1424The run time Configuration File
1425------------------------------
1426
1427. The format of the configuration file has changed. Instead of using "end" to
1428 terminate sections, it now uses "begin <name>" to start sections. This means
1429 that the sections, apart from the first, may appear in any order.
1430
1431. You can now include other files inside Exim run time configuration files, by
1432 using this syntax:
1433
1434 .include <file name>
1435
1436. Quotes round the file name are optional. Includes may be nested to any depth,
1437 but remember that Exim reads its configuration file often. The processing of
1438 .include happens early, at a physical line level, so, like comment lines, it
1439 can be used in the middle of an options setting, for example:
1440
1441 hosts_lookup = a.b.c \
1442 .include /some/file
1443
1444 Include processing happens _before_ macro processing. Its effect is simply to
1445 process the lines of the file as if they occurred inline where the .include
1446 appears.
1447
1448. A macro at the start of a configuration line can now turn the line into an
1449 empty line or a comment line. This applies to _logical_ input lines, that is,
1450 after any concatenations have been done.
1451
1452
1453Format of spool files
1454---------------------
1455
1456. -local_scan is used in spool files to record the value of $local_scan_data,
1457 the string returned from the locally-provided local_scan() function.
1458
1459
1460Renamed Options
1461---------------
1462
1463Some options have been renamed, to make their function clearer, or for
1464consistency.
1465
1466. receiver_unqualified_hosts has been renamed as recipient_unqualified_hosts.
1467 I'm going to use "recipient" everywhere in future.
1468
1469. helo_verify has become helo_verify_hosts.
1470
1471. remote_sort has become remote_sort_domains.
1472
1473. In the appendfile and pipe transports, "prefix" and "suffix" have become
1474 "message_prefix" and "message_suffix". In the generic router options,
1475 "prefix" and "suffix" have become "local_part_prefix" and "local_part_suffix".
1476
1477
1478Miscellaneous
1479-------------
1480
1481. ETRN serialization now uses a double fork, so that an Exim process (detached
1482 from the original input process) can wait for the command to finish. This
1483 means that it works whatever command ETRN causes to run. (Previously it
1484 worked only if ETRN ran "exim -Rxxx".)
1485
1486. For incoming messages, the server's port number is preserved, and is
1487 available in $interface_port. The privileged option -oMi can be used to
1488 set this value.
1489
1490. The -Mmd option (to mark addresses delivered) now operates in a
1491 case-sensitive manner.
1492
1493. Checks for duplicate deliveries are now case-sensitive in the local part.
1494
1495. The number of situations where Exim panics has been reduced. For example,
1496 expansion failures for the "domains" or "local_parts" options in a router now
1497 cause deferral instead of a panic.
1498
1499. EXPN no longer attempts to distinguish local and remote addresses (but you
1500 can cause it to be rejected for certain arguments in the ACL).
1501
1502. accept_timeout has been renamed as receive_timeout, to match
1503 smtp_receive_timeout.
1504
1505. The ability to check an ident value as part of an item in a host list has
1506 been removed.
1507
1508. The reject log shows a message's headers only if the rejection happens after
1509 the SMTP DATA command (because they aren't available for earlier checks). The
1510 sender, and up to five recipients are listed in Envelope-from: and
1511 Envelope-to: header lines. After the headers, a line of separator characters
1512 is output. Separators are no longer used for other reject log entries.
1513
1514. Because header checks are now done as part of ACLs, they now apply only to
1515 SMTP input.
1516
1517. The port number on SMTP connections is now logged in the format [aaaa]:ppp
1518 where aaaa is an IP address and ppp is a port, instead of in the format
1519 [aaaa.ppp] because the former format causes some software to complain about
1520 bad IP addresses.
1521
1522. The -oMa and -oMi options can now use the [aaaa]:ppp notation to set a port
1523 number, but they still also recognize the aaaa.ppp notation.
1524
1525. The build-time option HAVE_AUTH is abolished. Exim automatically includes
1526 authentication code if any authenticators are configured.
1527
1528. The nobody_user and nobody_group options have been abolished.
1529
1530. The $message_precedence variable has been abolished. The value is now
1531 available as $h_precedence:.
1532
1533. There's a new utility script called exim_checkaccess which packages up a call
1534 to Exim with the -bh option, for access control checking. The syntax is
1535
1536 exim_checkaccess <IP address> <email address> [exim options]
1537
1538 It runs "exim -bh <IP address>", does the SMTP dialogue, tests the result and
1539 outputs either "accepted" or "Rejected" and the SMTP response to the RCPT TO
1540 command. The sender is <> by default, but can be changed by the use of the
1541 -f option.
1542
1543. The default state of Exim is now to forbid domain literals. For this reason,
1544 the option that changes this has been renamed as allow_domain_literals.
1545
1546. The dns_check_names boolean option has been abolished. Checking is now turned
1547 off by unsetting dns_check_names_pattern.
1548
1549. The errors_address and freeze_tell_mailmaster options have been abolished. In
1550 their place there is a new option called freeze_tell, which can be set to a
1551 list of addresses. A message is sent to these addresses whenever a message is
1552 frozen - with the exception of failed bounce messages (this is not changed).
1553
1554. The message_size_limit_count_recipients option has been abolished on the
1555 grounds that it was a failed experiment.
1556
1557. The very-special-purpose X rewrite flag has been abolished. The facility it
1558 provided can now be done using the features of ACLs.
1559
1560. The timestamps_utc option has been abolished. The facility is now provided by
1561 setting timezone = utc.
1562
1563. The value of remote_max_parallel now defaults to 2.
1564
1565. ignore_errmsg_errors has been abolished. The effect can be achieved by
1566 setting ignore_bounce_errors_after = 0s. This option has been renamed from
1567 ignore_errmsg_errors_after to make its function clearer. The default value
1568 for ignore_bounce_errors_after is now 10w (10 weeks - i.e. likely to be
1569 longer than any other timeouts, thereby disabling the facility).
1570
1571. The default for message_size_limit is now 50M as a guard against DoS attacks.
1572
1573. The -qi option does only initial (first time) deliveries. This can be helpful
1574 if you are injecting message onto the queue using -odq and want a queue
1575 runner just to process new messages. You can also use -qqi if you want.
1576
1577. Rewriting and retry patterns are now anything that can be single address list
1578 items. They are processed by the same code, and are therefore expanded before
1579 the matching takes place. Regular expressions must be suitably quoted. These
1580 patterns may now be enclosed in double quotes so that white space may be
1581 included. Normal quote processing applies.
1582
1583. Some scripts were built in the util directory, which was a mistake, because
1584 they might be different for different platforms. Everything that is built is
1585 now built in the build directory. The util directory just contains a couple
1586 of scripts that are not modified at build time.
1587
1588. The installation script now installs the Exim binary as exim-v.vv-bb (where
1589 v.vv is the version number and bb is the build number), and points a symbolic
1590 link called "exim" to this binary. It does this in an atomic way so that
1591 there is no time when "exim" is non-existent. The script is clever enough to
1592 cope with an existing non-symbolic-link binary, converting it to the new
1593 scheme automatically (and atomically).
1594
1595. When installing utilities, Exim now uses cp instead of mv to add .O to the
1596 old ones, in order to preserve the permissions.
1597
1598. If the installation script is installing the default configuration, and
1599 /etc/aliases does not exist, the script installs a default version. This does
1600 not actually contain any aliases, but it does contain comments about ones
1601 that should be created. A warning is output to the user.
1602
1603. A delay warning message is not sent if all the addresses in a message get a
1604 "retry time not reached" error. Exim waits until a delivery is actually
1605 attempted, so as to be able to give a more informative message.
1606
1607. The existence of the three options deliver_load_max, queue_only_load, and
1608 deliver_queue_load_max was confusing, because their function overlapped. The
1609 first of them has been abolished. We are left with
1610
1611 queue_only_load no immediate delivery if load is high when
1612 message arrives
1613 deliver_queue_load_max no queued delivery if load is too high
1614
1615. The ability to edit message bodies (-Meb and the Eximon menu item) has been
1616 removed, on the grounds that it is bad practice to do this.
1617
1618. Eximstats is now Steve Campbell's patched version, which displays sizes in K
1619 and M and G, and can optionally generate HTML.
1620
1621. If bounce_sender_authentication is set to an email address, this address is
1622 used in an AUTH option of the MAIL command when sending bounce messages, if
1623 authentication is being used. For example, if you set
1624
1625 bounce_sender_authentication = mailer-daemon@your.domain
1626
1627 a bounce message will be sent over an authenticated connection using
1628
1629 MAIL FROM:<> AUTH=mailer-daemon@your.domain
1630
1631. untrusted_set_sender has changed from a boolean to an address pattern. It
1632 permits untrusted users to set sender addresses that match the pattern. Like
1633 all address patterns, it is expanded. The identity of the user is in
1634 $sender_ident, so you can, for example, restrict users to setting senders
1635 that start with their login ids by setting
1636
1637 untrusted_set_sender = ^$sender_ident-
1638
1639 The effect of the previous boolean can be achieved by setting the value to *.
1640 This option applies to all forms of local input.
1641
1642. The always_bcc option has been abolished. If an incoming message has no To:
1643 or Cc: headers, Exim now always adds an empty Bcc: line. This makes the
1644 message valid for RFC 822 (sic). In time, this can be removed, because RFC
1645 2822 does not require there to be a recipient header.
1646
1647. ACTION_OUTPUT=no is now the default in the Exim monitor.
1648
1649. dns_ipv4_lookup has changed from a boolean into a domain list, and it now
1650 applies only to those domains. Setting this option does not stop Exim from
1651 making IPv6 calls: if an MX lookup returns AAAA records, Exim will use them.
1652 What it does is to stop Exim looking for AAAA records explicitly.
1653
1654. The -G option is ignored (another Sendmail thing).
1655
1656. If no_bounce_return_message is set, the original message is not included in
1657 bounce messages. If you want to include additional information in the bounce
1658 message itself, you can use the existing errmsg_file and errmsg_text
1659 facilities.
1660
1661. -bdf runs the daemon in the foreground (i.e. not detached from the terminal),
1662 even when no debugging is requested.
1663
1664. Options for changing Exim's behaviour on receiving IPv4 options have been
1665 abolished. Exim now always refuses calls that set these options, and logs the
1666 incident. The abolished options are kill_ip_options, log_ip_options, and
1667 refuse_ip_options.
1668
1669. The pattern for each errors_copy entry is now matched as an item in an
1670 address list.
1671
1672. A number of options and variables that used the word "errmsg" have been
1673 changed to use "bounce" instead, because it seems that "bounce message" is
1674 now a reasonably well-understood term. I used it in the book and am now using
1675 it in the manual; it's a lot less cumbersome than "delivery error
1676 notification message". The changes are:
1677
1678 $errmsg_recipient => $bounce_recipient
1679 errmsg_file => bounce_message_file
1680 errmsg_text => bounce_message_text
1681 ignore_errmsg_errors_after => ignore_bounce_errors_after
1682
1683 For consistency, warnmsg_file has been changed to warn_message_file. However,
1684 the two variables $warnmsg_delay and $warnmsg_recipients are unchanged.
1685
1686 The hide_child_in_errmsg option has not changed, because it applies to both
1687 bounce and delay warning messages.
1688
1689. smtp_accept_max_per_host is now an expanded string, so it can be varied on
1690 a per-host basis. However, because this test happens in the daemon before it
1691 forks, the expansion should be kept as simple as possible (e.g. just inline
1692 tests of $sender_host_address).
1693
1694. The retry rules can now recognize the error "auth_failed", which happens when
1695 authentication is required, but cannot be done.
1696
1697. There's a new option called local_sender_retain which can be set if
1698 no_local_from_check is set. It causes Sender: headers to be retained in
1699 locally-submitted messages.
1700
1701. The -dropcr command line option now turns CRLF into LF, and leaves isolated
1702 CRs alone. Previously it simply dropped _all_ CR characters. There is now
1703 also a drop_cr main option which, if turned on, assumes -dropcr for all
1704 non-SMTP input.
1705
1706
1707Removal of Obsolete Things
1708--------------------------
1709
1710. The obsolete values "fail_soft" and "fail_hard" for the "self" option have
1711 been removed.
1712
1713. The obsolete "log" command has been removed from the filter language.
1714
1715. "service" was an obsolete synonym for "port" when specifying IP port numbers.
1716 It has been removed.
1717
1718. The obsolete option collapse_source_routes has been removed. It has done
1719 nothing since release 3.10.
1720
1721. The obsolete from_hack option in appendfile and pipe transports has been
1722 removed.
1723
1724. The obsolete ipv4_address_lookup has been abolished (dns_ipv4_lookup has been
1725 a synonym for some time, but it's changed - see above).
1726
1727. The obsolete generic transport options add_headers and remove_headers have
1728 been abolished. The new names, headers_add and headers_remove, have been
1729 available for some time.
1730
1731Philip Hazel
1732February 2002